{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "provenance": []
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "language_info": {
      "name": "python"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "source": [
        "**Introduction**\n",
        "\n",
        "Most PDF to text parsers do not provide layout information. Often times, even the sentences are split with arbritrary CR/LFs making it very difficult to find paragraph boundaries. This poses various challenges in chunking and adding long running contextual information such as section header to the passages while indexing/vectorizing PDFs for LLM applications such as retrieval augmented generation (RAG).\n",
        "\n",
        "LayoutPDFReader solves this problem by parsing PDFs along with hierarchical layout information such as:\n",
        "\n",
        "Sections and subsections along with their levels.\n",
        "Paragraphs - combines lines.\n",
        "Links between sections and paragraphs.\n",
        "Tables along with the section the tables are found in.\n",
        "Lists and nested lists.\n",
        "With LayoutPDFReader, developers can find optimal chunks of text to vectorize, and a solution for limited context window sizes of LLMs."
      ],
      "metadata": {
        "id": "2JULsCeQihR1"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "**Installation**\n",
        "\n",
        "Install the llmsherpa library."
      ],
      "metadata": {
        "id": "6ZQry7z6iCtO"
      }
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "0BwpEalxgvlN"
      },
      "outputs": [],
      "source": [
        "!pip install llmsherpa"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "The first step in using the LayoutPDFReader is to provide a url or file path to it and get back a document object."
      ],
      "metadata": {
        "id": "qy8mKt1NislJ"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "from llmsherpa.readers import LayoutPDFReader\n",
        "\n",
        "llmsherpa_api_url = \"https://readers.llmsherpa.com/api/document/developer/parseDocument?renderFormat=all\"\n",
        "pdf_url = \"https://arxiv.org/pdf/1910.13461.pdf\" # also allowed is a file path e.g. /home/downloads/xyz.pdf\n",
        "pdf_reader = LayoutPDFReader(llmsherpa_api_url)\n",
        "doc = pdf_reader.read_pdf(pdf_url)"
      ],
      "metadata": {
        "id": "wVCmWhxJiz9l"
      },
      "execution_count": 2,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "**Install LlamaIndex**\n",
        "\n",
        "In the following examples, we will use LlamaIndex for simplicity. Install the library if you haven't already."
      ],
      "metadata": {
        "id": "9YiiW2-Ni9HY"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "!pip install llama-index"
      ],
      "metadata": {
        "id": "tbVGDukQjdRd"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "**Setup OpenAI**\n",
        "\n",
        "Make sure your API Key is inserted."
      ],
      "metadata": {
        "id": "GqGIVd6qn6eC"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "import openai\n",
        "openai.api_key = #insert your api key here"
      ],
      "metadata": {
        "id": "3E73STaElAwN"
      },
      "execution_count": 7,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "**Summarize a Section using prompts**\n",
        "\n",
        "LayoutPDFReader offers powerful ways to pick sections and subsections from a large document and use LLMs to extract insights from a section.\n",
        "\n",
        "The following code looks for the Fine-tuning section of the document:"
      ],
      "metadata": {
        "id": "IY09Id11jXzp"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "from IPython.core.display import display, HTML\n",
        "selected_section = None\n",
        "# find a section in the document by title\n",
        "for section in doc.sections():\n",
        "    if section.title == '3 Fine-tuning BART':\n",
        "        selected_section = section\n",
        "        break\n",
        "# use include_children=True and recurse=True to fully expand the section.\n",
        "# include_children only returns at one sublevel of children whereas recurse goes through all the descendants\n",
        "HTML(section.to_html(include_children=True, recurse=True))"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 575
        },
        "id": "yOAON0CDj1mC",
        "outputId": "c3264379-a792-46cc-822c-03f4e08e3c53"
      },
      "execution_count": 4,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ],
            "text/html": [
              "<h3>3 Fine-tuning BART</h3><p>The representations produced by BART can be used in several ways for downstream applications.</p><h4>3.1 Sequence Classiﬁcation Tasks</h4><p>For sequence classiﬁcation tasks, the same input is fed into the encoder and decoder, and the ﬁnal hidden state of the ﬁnal decoder token is fed into new multi-class linear classiﬁer.\n",
              "This approach is related to the CLS token in BERT; however we add the additional token to the end so that representation for the token in the decoder can attend to decoder states from the complete input (Figure 3a).</p><h4>3.2 Token Classiﬁcation Tasks</h4><p>For token classiﬁcation tasks, such as answer endpoint classiﬁcation for SQuAD, we feed the complete document into the encoder and decoder, and use the top hidden state of the decoder as a representation for each word.\n",
              "This representation is used to classify the token.</p><h4>3.3 Sequence Generation Tasks</h4><p>Because BART has an autoregressive decoder, it can be directly ﬁne tuned for sequence generation tasks such as abstractive question answering and summarization.\n",
              "In both of these tasks, information is copied from the input but manipulated, which is closely related to the denoising pre-training objective.\n",
              "Here, the encoder input is the input sequence, and the decoder generates outputs autoregressively.</p><h4>3.4 Machine Translation</h4><p>We also explore using BART to improve machine translation decoders for translating into English.\n",
              "Previous work Edunov et al.\n",
              "(2019) has shown that models can be improved by incorporating pre-trained encoders, but gains from using pre-trained language models in decoders have been limited.\n",
              "We show that it is possible to use the entire BART model (both encoder and decoder) as a single pretrained decoder for machine translation, by adding a new set of encoder parameters that are learned from bitext (see Figure 3b).</p><p>More precisely, we replace BART’s encoder embedding layer with a new randomly initialized encoder.\n",
              "The model is trained end-to-end, which trains the new encoder to map foreign words into an input that BART can de-noise to English.\n",
              "The new encoder can use a separate vocabulary from the original BART model.</p><p>We train the source encoder in two steps, in both cases backpropagating the cross-entropy loss from the output of the BART model.\n",
              "In the ﬁrst step, we freeze most of BART parameters and only update the randomly initialized source encoder, the BART positional embeddings, and the self-attention input projection matrix of BART’s encoder ﬁrst layer.\n",
              "In the second step, we train all model parameters for a small number of iterations.</p>"
            ]
          },
          "metadata": {},
          "execution_count": 4
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "Now, let's create a custom summary of this text using a prompt:"
      ],
      "metadata": {
        "id": "1UnHxv2Ij-GO"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "from llama_index.llms import OpenAI\n",
        "context = selected_section.to_html(include_children=True, recurse=True)\n",
        "question = \"list all the tasks discussed and one line about each task\"\n",
        "resp = OpenAI().complete(f\"read this text and answer question: {question}:\\n{context}\")\n",
        "print(resp.text)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "F1-NOhTfkF3q",
        "outputId": "bd01f977-f56f-4103-af18-0c12141ef0db"
      },
      "execution_count": 9,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Tasks discussed in the text:\n",
            "\n",
            "1. Sequence Classification Tasks: The same input is fed into the encoder and decoder, and the final hidden state of the final decoder token is used for multi-class linear classification.\n",
            "2. Token Classification Tasks: The complete document is fed into the encoder and decoder, and the top hidden state of the decoder is used as a representation for each word to classify the token.\n",
            "3. Sequence Generation Tasks: BART can be fine-tuned for tasks like abstractive question answering and summarization, where the encoder input is the input sequence and the decoder generates outputs autoregressively.\n",
            "4. Machine Translation: BART can be used to improve machine translation decoders by incorporating pre-trained encoders and using the entire BART model as a single pretrained decoder. The new encoder is trained to map foreign words into an input that BART can de-noise to English.\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "**Analyze a Table using prompts**\n",
        "\n",
        "With LayoutPDFReader, you can iterate through all the tables in a document and use the power of LLMs to analyze a Table Let's look at the 6th table in this document. If you are using a notebook, you can display the table as follows:"
      ],
      "metadata": {
        "id": "cs6WO91KlOkF"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "from IPython.core.display import display, HTML\n",
        "HTML(doc.tables()[5].to_html())"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 147
        },
        "id": "Wdqvkig-lX6g",
        "outputId": "aca3c0ad-498b-415d-e434-66f645e9a010"
      },
      "execution_count": 10,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ],
            "text/html": [
              "<table><th><td colSpan=1></td><td colSpan=1>SQuAD 1.1 EM/F1</td><td colSpan=1>SQuAD 2.0 EM/F1</td><td colSpan=1>MNLI m/mm</td><td colSpan=1>SST Acc</td><td colSpan=1>QQP Acc</td><td colSpan=1>QNLI Acc</td><td colSpan=1>STS-B Acc</td><td colSpan=1>RTE Acc</td><td colSpan=1>MRPC Acc</td><td colSpan=1>CoLA Mcc</td></th><tr><td colSpan=1>BERT</td><td colSpan=1>84.1/90.9</td><td colSpan=1>79.0/81.8</td><td colSpan=1>86.6/-</td><td colSpan=1>93.2</td><td colSpan=1>91.3</td><td colSpan=1>92.3</td><td colSpan=1>90.0</td><td colSpan=1>70.4</td><td colSpan=1>88.0</td><td colSpan=1>60.6</td></tr><tr><td colSpan=1>UniLM</td><td colSpan=1>-/-</td><td colSpan=1>80.5/83.4</td><td colSpan=1>87.0/85.9</td><td colSpan=1>94.5</td><td colSpan=1>-</td><td colSpan=1>92.7</td><td colSpan=1>-</td><td colSpan=1>70.9</td><td colSpan=1>-</td><td colSpan=1>61.1</td></tr><tr><td colSpan=1>XLNet</td><td colSpan=1>89.0/94.5</td><td colSpan=1>86.1/88.8</td><td colSpan=1>89.8/-</td><td colSpan=1>95.6</td><td colSpan=1>91.8</td><td colSpan=1>93.9</td><td colSpan=1>91.8</td><td colSpan=1>83.8</td><td colSpan=1>89.2</td><td colSpan=1>63.6</td></tr><tr><td colSpan=1>RoBERTa</td><td colSpan=1>88.9/94.6</td><td colSpan=1>86.5/89.4</td><td colSpan=1>90.2/90.2</td><td colSpan=1>96.4</td><td colSpan=1>92.2</td><td colSpan=1>94.7</td><td colSpan=1>92.4</td><td colSpan=1>86.6</td><td colSpan=1>90.9</td><td colSpan=1>68.0</td></tr><tr><td colSpan=1>BART</td><td colSpan=1>88.8/94.6</td><td colSpan=1>86.1/89.2</td><td colSpan=1>89.9/90.1</td><td colSpan=1>96.6</td><td colSpan=1>92.5</td><td colSpan=1>94.9</td><td colSpan=1>91.2</td><td colSpan=1>87.0</td><td colSpan=1>90.4</td><td colSpan=1>62.8</td></tr></table>"
            ]
          },
          "metadata": {},
          "execution_count": 10
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "Now let's ask a question to analyze this table:"
      ],
      "metadata": {
        "id": "6NEl4vzvlg5W"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "from llama_index.llms import OpenAI\n",
        "context = doc.tables()[5].to_html()\n",
        "resp = OpenAI().complete(f\"read this table and answer question: which model has the best performance on squad 2.0:\\n{context}\")\n",
        "print(resp.text)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "N7oHRsbcloL6",
        "outputId": "4b5b46df-bc01-49a2-ad75-dca4da50132a"
      },
      "execution_count": 11,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "The model with the best performance on SQuAD 2.0 is RoBERTa, with an EM/F1 score of 86.5/89.4.\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "That's it! LayoutPDFReader also supports tables with nested headers and header rows.\n",
        "\n",
        "Here's an example with nested headers (note that the HTML doesn't render properly in ipython but the html structure is correct):"
      ],
      "metadata": {
        "id": "eFefOzQ9ltM3"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "from IPython.core.display import display, HTML\n",
        "HTML(doc.tables()[6].to_html())"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 190
        },
        "id": "SShVijKPlz36",
        "outputId": "e4435b65-1900-43b2-ac8f-0d7d42f95a43"
      },
      "execution_count": 12,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ],
            "text/html": [
              "<table><th><td colSpan=1></td><td>CNN/DailyMail</td><td>XSum</td></th><th><td colSpan=1></td><td colSpan=1>R1</td><td colSpan=1>R2</td><td colSpan=1>RL</td><td colSpan=1>R1</td><td colSpan=1>R2</td><td colSpan=1>RL</td></th><tr><td colSpan=1>Lead-3</td><td colSpan=1>40.42</td><td colSpan=1>17.62</td><td colSpan=1>36.67</td><td colSpan=1>16.30</td><td colSpan=1>1.60</td><td colSpan=1>11.95</td></tr><tr><td colSpan=1>PTGEN (See et al., 2017)</td><td colSpan=1>36.44</td><td colSpan=1>15.66</td><td colSpan=1>33.42</td><td colSpan=1>29.70</td><td colSpan=1>9.21</td><td colSpan=1>23.24</td></tr><tr><td colSpan=1>PTGEN+COV (See et al., 2017)</td><td colSpan=1>39.53</td><td colSpan=1>17.28</td><td colSpan=1>36.38</td><td colSpan=1>28.10</td><td colSpan=1>8.02</td><td colSpan=1>21.72</td></tr><tr><td colSpan=1>UniLM</td><td colSpan=1>43.33</td><td colSpan=1>20.21</td><td colSpan=1>40.51</td><td colSpan=1>-</td><td colSpan=1>-</td><td colSpan=1>-</td></tr><tr><td colSpan=1>BERTSUMABS (Liu & Lapata, 2019)</td><td colSpan=1>41.72</td><td colSpan=1>19.39</td><td colSpan=1>38.76</td><td colSpan=1>38.76</td><td colSpan=1>16.33</td><td colSpan=1>31.15</td></tr><tr><td colSpan=1>BERTSUMEXTABS (Liu & Lapata, 2019)</td><td colSpan=1>42.13</td><td colSpan=1>19.60</td><td colSpan=1>39.18</td><td colSpan=1>38.81</td><td colSpan=1>16.50</td><td colSpan=1>31.27</td></tr><tr><td colSpan=1>BART</td><td colSpan=1>44.16</td><td colSpan=1>21.28</td><td colSpan=1>40.90</td><td colSpan=1>45.14</td><td colSpan=1>22.27</td><td colSpan=1>37.25</td></tr></table>"
            ]
          },
          "metadata": {},
          "execution_count": 12
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "Now let's ask an interesting question:"
      ],
      "metadata": {
        "id": "7F0GF1qNmJlS"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "from llama_index.llms import OpenAI\n",
        "context = doc.tables()[6].to_html()\n",
        "question = \"tell me about R1 of bart for different datasets\"\n",
        "resp = OpenAI().complete(f\"read this table and answer question: {question}:\\n{context}\")\n",
        "print(resp.text)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "WsUCGpm3mRJJ",
        "outputId": "65c5136e-40a5-4ea1-a55b-b36eedefe050"
      },
      "execution_count": 13,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "R1 of BART for different datasets:\n",
            "\n",
            "- For the CNN/DailyMail dataset, the R1 score of BART is 44.16.\n",
            "- For the XSum dataset, the R1 score of BART is 21.28.\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "\n",
        "**Vector search and Retrieval Augmented Generation with Smart Chunking**\n",
        "\n",
        "LayoutPDFReader does smart chunking keeping the integrity of related text together:\n",
        "\n",
        "All list items are together including the paragraph that precedes the list.\n",
        "Items in a table are chuncked together\n",
        "Contextual information from section headers and nested section headers is included\n",
        "The following code creates a LlamaIndex query engine from LayoutPDFReader document chunks"
      ],
      "metadata": {
        "id": "wE7I3L7XmWl3"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "from llama_index.readers.schema.base import Document\n",
        "from llama_index import VectorStoreIndex\n",
        "\n",
        "index = VectorStoreIndex([])\n",
        "for chunk in doc.chunks():\n",
        "    index.insert(Document(text=chunk.to_context_text(), extra_info={}))\n",
        "query_engine = index.as_query_engine()"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "YGqsLP5zmirK",
        "outputId": "cb521786-ef48-4e95-8577-5a08e2c59b74"
      },
      "execution_count": 14,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "[nltk_data] Downloading package punkt to /tmp/llama_index...\n",
            "[nltk_data]   Unzipping tokenizers/punkt.zip.\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "Let's run one query:"
      ],
      "metadata": {
        "id": "0f797GR6nGKw"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "response = query_engine.query(\"list all the tasks that work with bart\")\n",
        "print(response)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "vC9H9c38nOsG",
        "outputId": "bbbe1616-1765-41d9-fd71-997b9b380363"
      },
      "execution_count": 15,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "BART works well for text generation, comprehension tasks, abstractive dialogue, question answering, and summarization tasks.\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "Let's try another query that needs answer from a table:"
      ],
      "metadata": {
        "id": "8ylioaL0nUyr"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "response = query_engine.query(\"what is the bart performance score on squad\")\n",
        "print(response)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "DZEHBSsnnc30",
        "outputId": "db684d29-8757-4180-a922-ab7a14d07567"
      },
      "execution_count": 16,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "The BART performance score on SQuAD is 88.8 for Exact Match (EM) and 94.6 for F1 score.\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "**Get the Raw JSON**\n",
        "\n",
        "To get the complete json returned by llmsherpa service and process it differently, simply get the json attribute"
      ],
      "metadata": {
        "id": "YT67xijEnh1X"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "doc.json"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "_YHwKINfnri7",
        "outputId": "b676008e-2b25-40b8-d4fd-0e5bf6ba47df"
      },
      "execution_count": 17,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "[{'block_class': 'cls_0',\n",
              "  'block_idx': 0,\n",
              "  'level': 0,\n",
              "  'page_idx': 0,\n",
              "  'sentences': ['BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension'],\n",
              "  'tag': 'header'},\n",
              " {'block_class': 'cls_1',\n",
              "  'block_idx': 1,\n",
              "  'level': 0,\n",
              "  'page_idx': 0,\n",
              "  'sentences': ['Mike Lewis*, Yinhan Liu*, Naman Goyal*, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Ves Stoyanov, Luke Zettlemoyer Facebook AI'],\n",
              "  'tag': 'para'},\n",
              " {'block_class': 'cls_5',\n",
              "  'block_idx': 2,\n",
              "  'level': 1,\n",
              "  'page_idx': 0,\n",
              "  'sentences': ['{mikelewis,yinhanliu,naman}@fb.com'],\n",
              "  'tag': 'header'},\n",
              " {'block_class': 'cls_1',\n",
              "  'block_idx': 3,\n",
              "  'level': 2,\n",
              "  'page_idx': 0,\n",
              "  'sentences': ['Abstract'],\n",
              "  'tag': 'header'},\n",
              " {'block_class': 'cls_7',\n",
              "  'block_idx': 4,\n",
              "  'level': 3,\n",
              "  'page_idx': 0,\n",
              "  'sentences': ['We present BART, a denoising autoencoder for pretraining sequence-to-sequence models.',\n",
              "   'BART is trained by (1) corrupting text with an arbitrary noising function, and (2) learning a model to reconstruct the original text.',\n",
              "   'It uses a standard Tranformer-based neural machine translation architecture which, despite its simplicity, can be seen as generalizing BERT (due to the bidirectional encoder), GPT (with the left-to-right decoder), and many other more recent pretraining schemes.',\n",
              "   'We evaluate a number of noising approaches, ﬁnding the best performance by both randomly shufﬂing the order of the original sentences and using a novel in-ﬁlling scheme, where spans of text are replaced with a single mask token.',\n",
              "   'BART is particularly effective when ﬁne tuned for text generation but also works well for comprehension tasks.',\n",
              "   'It matches the performance of RoBERTa with comparable training resources on GLUE and SQuAD, achieves new stateof-the-art results on a range of abstractive dialogue, question answering, and summarization tasks, with gains of up to 6 ROUGE.',\n",
              "   'BART also provides a 1.1 BLEU increase over a back-translation system for machine translation, with only target language pretraining.',\n",
              "   'We also report ablation experiments that replicate other pretraining schemes within the BART framework, to better measure which factors most inﬂuence end-task performance.'],\n",
              "  'tag': 'para'},\n",
              " {'block_class': 'cls_1',\n",
              "  'block_idx': 5,\n",
              "  'level': 2,\n",
              "  'page_idx': 0,\n",
              "  'sentences': ['1 Introduction'],\n",
              "  'tag': 'header'},\n",
              " {'block_class': 'cls_7',\n",
              "  'block_idx': 6,\n",
              "  'level': 3,\n",
              "  'page_idx': 0,\n",
              "  'sentences': ['methods have achieved remarkable success in a wide range of NLP tasks (Mikolov et al., 2013; Peters et al., 2018; Devlin et al., 2019; Joshi et al., 2019; Yang et al., 2019; Liu et al., 2019).',\n",
              "   'The most successful approaches have been variants of masked language models, which are denoising autoencoders that are trained to reconstruct text where a random subset of the words has been masked out.',\n",
              "   'Recent work has shown gains by improving the distribution of masked tokens (Joshi et al., 2019), the order in which masked tokens are predicted (Yang et al., 2019), and the available context for replacing masked tokens (Dong et al., 2019).',\n",
              "   'However, these methods typically focus on particular types of end tasks (e.g. span prediction, generation, etc.), limiting their applicability.'],\n",
              "  'tag': 'para'},\n",
              " {'block_class': 'cls_7',\n",
              "  'block_idx': 7,\n",
              "  'level': 3,\n",
              "  'page_idx': 0,\n",
              "  'sentences': ['In this paper, we present BART, which pre-trains a model combining Bidirectional and Auto-Regressive Transformers.',\n",
              "   'BART is a denoising autoencoder built with a sequence-to-sequence model that is applicable to a very wide range of end tasks.',\n",
              "   'Pretraining has two stages (1) text is corrupted with an arbitrary noising function, and (2) a sequence-to-sequence model is learned to reconstruct the original text.',\n",
              "   'BART uses a standard Tranformer-based neural machine translation architecture which, despite its simplicity, can be seen as generalizing BERT (due to the bidirectional encoder), GPT (with the left-to-right decoder), and many other more recent pretraining schemes (see Figure 1).'],\n",
              "  'tag': 'para'},\n",
              " {'block_class': 'cls_7',\n",
              "  'block_idx': 8,\n",
              "  'level': 3,\n",
              "  'page_idx': 0,\n",
              "  'sentences': ['A key advantage of this setup is the noising ﬂexibility; arbitrary transformations can be applied to the original text, including changing its length.',\n",
              "   'We evaluate a number of noising approaches, ﬁnding the best performance by both randomly shufﬂing the order of the original sentences and using a novel in-ﬁlling scheme, where arbitrary length spans of text (including zero length) are replaced with a single mask token.',\n",
              "   'This approach generalizes the original word masking and next sentence prediction objectives in BERT by forcing the model to reason more about overall sentence length and make longer range transformations to the input.'],\n",
              "  'tag': 'para'},\n",
              " {'block_class': 'cls_7',\n",
              "  'block_idx': 9,\n",
              "  'level': 3,\n",
              "  'page_idx': 0,\n",
              "  'sentences': ['BART is particularly effective when ﬁne tuned for text generation but also works well for comprehension tasks.',\n",
              "   'It matches the performance of RoBERTa (Liu et al., 2019) with comparable training resources on GLUE (Wang et al., 2018) and SQuAD (Rajpurkar et al., 2016), and achieves new state-of-the-art results on a range of abstractive dialogue, question answering, and summarization tasks.',\n",
              "   'For example, it improves performance by 6 ROUGE over previous work on XSum (Narayan et al., 2018).'],\n",
              "  'tag': 'para'},\n",
              " {'block_class': 'cls_7',\n",
              "  'block_idx': 10,\n",
              "  'level': 3,\n",
              "  'page_idx': 0,\n",
              "  'sentences': ['BART also opens up new ways of thinking about ﬁne tuning.',\n",
              "   'We present a new scheme for machine translation where a BART model is stacked above a few additional transformer layers.',\n",
              "   'These layers are trained to essentially translate the foreign language to noised'],\n",
              "  'tag': 'para'},\n",
              " {'block_class': 'cls_10',\n",
              "  'block_idx': 11,\n",
              "  'level': 3,\n",
              "  'page_idx': 1,\n",
              "  'sentences': ['B D A B C D E'],\n",
              "  'tag': 'header'},\n",
              " {'block_class': 'cls_10',\n",
              "  'block_idx': 12,\n",
              "  'level': 4,\n",
              "  'page_idx': 1,\n",
              "  'sentences': ['Autoregressive Decoder'],\n",
              "  'tag': 'para'},\n",
              " {'block_class': 'cls_8',\n",
              "  'block_idx': 13,\n",
              "  'level': 3,\n",
              "  'page_idx': 1,\n",
              "  'sentences': ['Bidirectional Encoder'],\n",
              "  'tag': 'header'},\n",
              " {'block_class': 'cls_8',\n",
              "  'block_idx': 14,\n",
              "  'level': 4,\n",
              "  'page_idx': 1,\n",
              "  'sentences': ['A _ C _ E'],\n",
              "  'tag': 'header'},\n",
              " {'block_class': 'cls_10',\n",
              "  'block_idx': 15,\n",
              "  'level': 5,\n",
              "  'page_idx': 1,\n",
              "  'sentences': ['<s> A B C D'],\n",
              "  'tag': 'header'},\n",
              " {'block_class': 'cls_9',\n",
              "  'block_idx': 16,\n",
              "  'level': 6,\n",
              "  'page_idx': 1,\n",
              "  'sentences': ['(a) BERT: Random tokens are replaced with masks, and the document is encoded bidirectionally.',\n",
              "   'Missing tokens are predicted independently, so BERT cannot easily be (b) GPT: Tokens are predicted auto-regressively, meaning GPT can be used for generation.',\n",
              "   'However words can only condition on leftward context, so it cannot learn bidirec- tional interactions.'],\n",
              "  'tag': 'list_item'},\n",
              " {'block_class': 'cls_9',\n",
              "  'block_idx': 17,\n",
              "  'level': 6,\n",
              "  'page_idx': 1,\n",
              "  'sentences': ['used for generation.'],\n",
              "  'tag': 'para'},\n",
              " {'block_class': 'cls_10',\n",
              "  'block_idx': 18,\n",
              "  'level': 6,\n",
              "  'page_idx': 1,\n",
              "  'sentences': ['A B C D E'],\n",
              "  'tag': 'header'},\n",
              " {'block_class': 'cls_10',\n",
              "  'block_idx': 19,\n",
              "  'left': 211.35,\n",
              "  'level': 7,\n",
              "  'name': 'A B C D E',\n",
              "  'page_idx': 1,\n",
              "  'table_rows': [{'block_idx': 19,\n",
              "    'cells': [{'cell_value': 'Bidirectional Encoder'},\n",
              "     {'cell_value': 'Autoregressive Decoder'}],\n",
              "    'type': 'table_data_row'},\n",
              "   {'block_idx': 20,\n",
              "    'cell_value': 'A _ B _ E <s> A B C D',\n",
              "    'col_span': 2,\n",
              "    'type': 'full_row'}],\n",
              "  'tag': 'table',\n",
              "  'top': 226.4},\n",
              " {'block_class': 'cls_9',\n",
              "  'block_idx': 21,\n",
              "  'level': 6,\n",
              "  'page_idx': 1,\n",
              "  'sentences': ['(c) BART: Inputs to the encoder need not be aligned with decoder outputs, allowing arbitary noise transformations.',\n",
              "   'Here, a document has been corrupted by replacing spans of text with mask symbols.',\n",
              "   'The corrupted document (left) is encoded with a bidirectional model, and then the likelihood of the original document (right) is calculated with an autoregressive decoder.',\n",
              "   'For ﬁne-tuning, an uncorrupted document is input to both the encoder and decoder, and we use representations from the ﬁnal hidden state of the decoder.'],\n",
              "  'tag': 'list_item'},\n",
              " {'block_class': 'cls_7',\n",
              "  'block_idx': 22,\n",
              "  'level': 6,\n",
              "  'page_idx': 1,\n",
              "  'sentences': ['Figure 1: A schematic comparison of BART with BERT (Devlin et al., 2019) and GPT (Radford et al., 2018).'],\n",
              "  'tag': 'para'},\n",
              " {'block_class': 'cls_7',\n",
              "  'block_idx': 23,\n",
              "  'level': 6,\n",
              "  'page_idx': 1,\n",
              "  'sentences': ['English, by propagation through BART, thereby using BART as a pre-trained target-side language model.',\n",
              "   'This approach improves performance over a strong back-translation MT baseline by 1.1 BLEU on the WMT Romanian-English benchmark.'],\n",
              "  'tag': 'para'},\n",
              " {'block_class': 'cls_7',\n",
              "  'block_idx': 24,\n",
              "  'level': 6,\n",
              "  'page_idx': 1,\n",
              "  'sentences': ['To better understand these effects, we also report an ablation analysis that replicates other recently proposed training objectives.',\n",
              "   'This study allows us to carefully control for a number of factors, including data and optimization parameters, which have been shown to be as important for overall performance as the selection of training objectives (Liu et al., 2019).',\n",
              "   'We ﬁnd that BART exhibits the most consistently strong performance across the full range of tasks we consider.'],\n",
              "  'tag': 'para'},\n",
              " {'block_class': 'cls_1',\n",
              "  'block_idx': 25,\n",
              "  'level': 2,\n",
              "  'page_idx': 1,\n",
              "  'sentences': ['2 Model'],\n",
              "  'tag': 'header'},\n",
              " {'block_class': 'cls_7',\n",
              "  'block_idx': 26,\n",
              "  'level': 3,\n",
              "  'page_idx': 1,\n",
              "  'sentences': ['is a denoising autoencoder that maps a corrupted document to the original document it was derived from.',\n",
              "   'It is implemented as a sequence-to-sequence model with a bidirectional encoder over corrupted text and a left-to-right autoregressive decoder.',\n",
              "   'For pre-training, we optimize the negative log likelihood of the original document.'],\n",
              "  'tag': 'para'},\n",
              " {'block_class': 'cls_11',\n",
              "  'block_idx': 27,\n",
              "  'level': 3,\n",
              "  'page_idx': 1,\n",
              "  'sentences': ['2.1 Architecture'],\n",
              "  'tag': 'header'},\n",
              " {'block_class': 'cls_7',\n",
              "  'block_idx': 28,\n",
              "  'level': 4,\n",
              "  'page_idx': 1,\n",
              "  'sentences': ['BART uses the standard sequence-to-sequence Transformer architecture from (Vaswani et al., 2017), except, following GPT, that we modify ReLU activation functions to GeLUs (Hendrycks & Gimpel, 2016) and initialise parameters from N (0, 0.02).',\n",
              "   'For our base model, we use 6 layers in the encoder and de- coder, and for our large model we use 12 layers in each.',\n",
              "   'The architecture is closely related to that used in BERT, with the following differences: (1) each layer of the decoder additionally performs cross-attention over the ﬁnal hidden layer of the encoder (as in the transformer sequence-to-sequence model); and (2) BERT uses an additional feed-forward network before wordprediction, which BART does not.',\n",
              "   'In total, BART contains roughly 10% more parameters than the equivalently sized BERT model.'],\n",
              "  'tag': 'para'},\n",
              " {'block_class': 'cls_11',\n",
              "  'block_idx': 29,\n",
              "  'level': 3,\n",
              "  'page_idx': 1,\n",
              "  'sentences': ['2.2 Pre-training BART'],\n",
              "  'tag': 'header'},\n",
              " {'block_class': 'cls_7',\n",
              "  'block_idx': 30,\n",
              "  'level': 4,\n",
              "  'page_idx': 1,\n",
              "  'sentences': ['BART is trained by corrupting documents and then optimizing a reconstruction loss—the cross-entropy between the decoder’s output and the original document.',\n",
              "   'Unlike existing denoising autoencoders, which are tailored to speciﬁc noising schemes, BART allows us to apply any type of document corruption.',\n",
              "   'In the extreme case, where all information about the source is lost, BART is equivalent to a language model.'],\n",
              "  'tag': 'para'},\n",
              " {'block_class': 'cls_7',\n",
              "  'block_idx': 31,\n",
              "  'level': 4,\n",
              "  'page_idx': 1,\n",
              "  'sentences': ['We experiment with several previously proposed and novel transformations, but we believe there is a signiﬁcant potential for development of other new alternatives.',\n",
              "   'The transformations we used are summarized below, and examples are shown in Figure 2.'],\n",
              "  'tag': 'para'},\n",
              " {'block_class': 'cls_7',\n",
              "  'block_idx': 32,\n",
              "  'level': 4,\n",
              "  'page_idx': 1,\n",
              "  'sentences': ['Token Masking Following BERT (Devlin et al., 2019), random tokens are sampled and replaced with [MASK] elements.'],\n",
              "  'tag': 'para'},\n",
              " {'block_class': 'cls_7',\n",
              "  'block_idx': 33,\n",
              "  'level': 4,\n",
              "  'page_idx': 1,\n",
              "  'sentences': ['Token Deletion Random tokens are deleted from the input.',\n",
              "   'In contrast to token masking, the model must decide which positions are missing inputs.'],\n",
              "  'tag': 'para'},\n",
              " {'block_class': 'cls_16',\n",
              "  'block_idx': 34,\n",
              "  'left': 152.36,\n",
              "  'level': 3,\n",
              "  'name': '2.2 Pre-training BART',\n",
              "  'page_idx': 2,\n",
              "  'table_rows': [{'block_idx': 34,\n",
              "    'cells': [{'cell_value': 'A _C . _ E .'},\n",
              "     {'cell_value': 'D E . A B C .'},\n",
              "     {'cell_value': 'C . D E . A B'}],\n",
              "    'type': 'table_header'},\n",
              "   {'block_idx': 35,\n",
              "    'cells': [{'cell_value': ''},\n",
              "     {'cell_value': 'Sentence Permutation'},\n",
              "     {'cell_value': 'Document RotationToken Masking'}],\n",
              "    'type': 'table_data_row'},\n",
              "   {'block_idx': 36,\n",
              "    'cells': [{'cell_value': ''},\n",
              "     {'cell_value': 'A B C . D E .A . C . E .'},\n",
              "     {'cell_value': 'A _ . D _ E .'}],\n",
              "    'type': 'table_data_row'}],\n",
              "  'tag': 'table',\n",
              "  'top': 65.33},\n",
              " {'block_class': 'cls_16',\n",
              "  'block_idx': 37,\n",
              "  'level': 3,\n",
              "  'page_idx': 2,\n",
              "  'sentences': ['Token Deletion Text Inﬁlling'],\n",
              "  'tag': 'header'},\n",
              " {'block_class': 'cls_7',\n",
              "  'block_idx': 38,\n",
              "  'level': 4,\n",
              "  'page_idx': 2,\n",
              "  'sentences': ['Figure 2: Transformations for noising the input that we experiment with.',\n",
              "   'These transformations can be composed.'],\n",
              "  'tag': 'para'},\n",
              " {'block_class': 'cls_7',\n",
              "  'block_idx': 39,\n",
              "  'level': 4,\n",
              "  'page_idx': 2,\n",
              "  'sentences': ['Text Inﬁlling A number of text spans are sampled, with span lengths drawn from a Poisson distribution (λ = 3).',\n",
              "   'Each span is replaced with a single [MASK] token.',\n",
              "   '0-length spans correspond to the insertion of [MASK] tokens.',\n",
              "   'Text inﬁlling is inspired by SpanBERT (Joshi et al., 2019), but SpanBERT samples span lengths from a different (clamped geometric) distribution, and replaces each span with a sequence of [MASK] tokens of exactly the same length.',\n",
              "   'Text inﬁlling teaches the model to predict how many tokens are missing from a span.'],\n",
              "  'tag': 'para'},\n",
              " {'block_class': 'cls_7',\n",
              "  'block_idx': 40,\n",
              "  'level': 4,\n",
              "  'page_idx': 2,\n",
              "  'sentences': ['Sentence Permutation A document is divided into sentences based on full stops, and these sentences are shufﬂed in a random order.'],\n",
              "  'tag': 'para'},\n",
              " {'block_class': 'cls_7',\n",
              "  'block_idx': 41,\n",
              "  'level': 4,\n",
              "  'page_idx': 2,\n",
              "  'sentences': ['Document Rotation A token is chosen uniformly at random, and the document is rotated so that it begins with that token.',\n",
              "   'This task trains the model to identify the start of the document.'],\n",
              "  'tag': 'para'},\n",
              " {'block_class': 'cls_1',\n",
              "  'block_idx': 42,\n",
              "  'level': 2,\n",
              "  'page_idx': 2,\n",
              "  'sentences': ['3 Fine-tuning BART'],\n",
              "  'tag': 'header'},\n",
              " {'block_class': 'cls_7',\n",
              "  'block_idx': 43,\n",
              "  'level': 3,\n",
              "  'page_idx': 2,\n",
              "  'sentences': ['The representations produced by BART can be used in several ways for downstream applications.'],\n",
              "  'tag': 'para'},\n",
              " {'block_class': 'cls_11',\n",
              "  'block_idx': 44,\n",
              "  'level': 3,\n",
              "  'page_idx': 2,\n",
              "  'sentences': ['3.1 Sequence Classiﬁcation Tasks'],\n",
              "  'tag': 'header'},\n",
              " {'block_class': 'cls_7',\n",
              "  'block_idx': 45,\n",
              "  'level': 3,\n",
              "  'page_idx': 2,\n",
              "  'sentences': ['For sequence classiﬁcation tasks, the same input is fed into the encoder and decoder, and the ﬁnal hidden state of the ﬁnal decoder token is fed into new multi-class linear classiﬁer.',\n",
              "   'This approach is related to the CLS token in BERT; however we add the additional token to the end so that representation for the token in the decoder can attend to decoder states from the complete input (Figure 3a).'],\n",
              "  'tag': 'para'},\n",
              " {'block_class': 'cls_11',\n",
              "  'block_idx': 46,\n",
              "  'level': 3,\n",
              "  'page_idx': 2,\n",
              "  'sentences': ['3.2 Token Classiﬁcation Tasks'],\n",
              "  'tag': 'header'},\n",
              " {'block_class': 'cls_7',\n",
              "  'block_idx': 47,\n",
              "  'level': 5,\n",
              "  'page_idx': 2,\n",
              "  'sentences': ['For token classiﬁcation tasks, such as answer endpoint classiﬁcation for SQuAD, we feed the complete document into the encoder and decoder, and use the top hidden state of the decoder as a representation for each word.',\n",
              "   'This representation is used to classify the token.'],\n",
              "  'tag': 'para'},\n",
              " {'block_class': 'cls_11',\n",
              "  'block_idx': 48,\n",
              "  'level': 3,\n",
              "  'page_idx': 2,\n",
              "  'sentences': ['3.3 Sequence Generation Tasks'],\n",
              "  'tag': 'header'},\n",
              " {'block_class': 'cls_7',\n",
              "  'block_idx': 49,\n",
              "  'level': 5,\n",
              "  'page_idx': 2,\n",
              "  'sentences': ['Because BART has an autoregressive decoder, it can be directly ﬁne tuned for sequence generation tasks such as abstractive question answering and summarization.',\n",
              "   'In both of these tasks, information is copied from the input but manipulated, which is closely related to the denoising pre-training objective.',\n",
              "   'Here, the encoder input is the input sequence, and the decoder generates outputs autoregressively.'],\n",
              "  'tag': 'para'},\n",
              " {'block_class': 'cls_11',\n",
              "  'block_idx': 50,\n",
              "  'level': 3,\n",
              "  'page_idx': 2,\n",
              "  'sentences': ['3.4 Machine Translation'],\n",
              "  'tag': 'header'},\n",
              " {'block_class': 'cls_7',\n",
              "  'block_idx': 51,\n",
              "  'level': 5,\n",
              "  'page_idx': 2,\n",
              "  'sentences': ['We also explore using BART to improve machine translation decoders for translating into English.',\n",
              "   'Previous work Edunov et al.',\n",
              "   '(2019) has shown that models can be improved by incorporating pre-trained encoders, but gains from using pre-trained language models in decoders have been limited.',\n",
              "   'We show that it is possible to use the entire BART model (both encoder and decoder) as a single pretrained decoder for machine translation, by adding a new set of encoder parameters that are learned from bitext (see Figure 3b).'],\n",
              "  'tag': 'para'},\n",
              " {'block_class': 'cls_7',\n",
              "  'block_idx': 52,\n",
              "  'level': 5,\n",
              "  'page_idx': 2,\n",
              "  'sentences': ['More precisely, we replace BART’s encoder embedding layer with a new randomly initialized encoder.',\n",
              "   'The model is trained end-to-end, which trains the new encoder to map foreign words into an input that BART can de-noise to English.',\n",
              "   'The new encoder can use a separate vocabulary from the original BART model.'],\n",
              "  'tag': 'para'},\n",
              " {'block_class': 'cls_7',\n",
              "  'block_idx': 53,\n",
              "  'level': 5,\n",
              "  'page_idx': 2,\n",
              "  'sentences': ['We train the source encoder in two steps, in both cases backpropagating the cross-entropy loss from the output of the BART model.',\n",
              "   'In the ﬁrst step, we freeze most of BART parameters and only update the randomly initialized source encoder, the BART positional embeddings, and the self-attention input projection matrix of BART’s encoder ﬁrst layer.',\n",
              "   'In the second step, we train all model parameters for a small number of iterations.'],\n",
              "  'tag': 'para'},\n",
              " {'block_class': 'cls_1',\n",
              "  'block_idx': 54,\n",
              "  'level': 2,\n",
              "  'page_idx': 2,\n",
              "  'sentences': ['4 Comparing Pre-training Objectives'],\n",
              "  'tag': 'header'},\n",
              " {'block_class': 'cls_7',\n",
              "  'block_idx': 55,\n",
              "  'level': 3,\n",
              "  'page_idx': 2,\n",
              "  'sentences': ['BART supports a much wider range of noising schemes during pre-training than previous work.',\n",
              "   'We compare a range of options using base-size models (6 encoder and 6 decoder layers, with a hidden size of 768), evaluated on a representative subset of the tasks we will consider for the full large scale experiments in §5.'],\n",
              "  'tag': 'para'},\n",
              " {'block_class': 'cls_11',\n",
              "  'block_idx': 56,\n",
              "  'level': 3,\n",
              "  'page_idx': 2,\n",
              "  'sentences': ['4.1 Comparison Objectives'],\n",
              "  'tag': 'header'},\n",
              " {'block_class': 'cls_7',\n",
              "  'block_idx': 57,\n",
              "  'level': 3,\n",
              "  'page_idx': 2,\n",
              "  'sentences': ['While many pre-training objectives have been proposed, fair comparisons between these have been difﬁcult to perform, at least in part due to differences in training data, training resources, architectural differences between models, and ﬁne-tuning procedures.',\n",
              "   'We'],\n",
              "  'tag': 'para'},\n",
              " {'block_class': 'cls_17',\n",
              "  'block_idx': 58,\n",
              "  'level': 4,\n",
              "  'page_idx': 3,\n",
              "  'sentences': ['A', 'B', 'C', 'D E'],\n",
              "  'tag': 'para'},\n",
              " {'block_class': 'cls_8',\n",
              "  'block_idx': 59,\n",
              "  'level': 3,\n",
              "  'page_idx': 3,\n",
              "  'sentences': ['label'],\n",
              "  'tag': 'header'},\n",
              " {'block_class': 'cls_17',\n",
              "  'block_idx': 60,\n",
              "  'left': 350.41,\n",
              "  'level': 4,\n",
              "  'name': 'label',\n",
              "  'page_idx': 3,\n",
              "  'table_rows': [{'block_idx': 60,\n",
              "    'cells': [{'cell_value': 'Pre-trained Encoder'},\n",
              "     {'cell_value': 'Pre-trained Decoder'}],\n",
              "    'type': 'table_header'},\n",
              "   {'block_idx': 61,\n",
              "    'cells': [{'cell_value': 'Pre-trained Encoder'},\n",
              "     {'cell_value': 'Pre-trained Decoder'}],\n",
              "    'type': 'table_data_row'}],\n",
              "  'tag': 'table',\n",
              "  'top': 82.59},\n",
              " {'block_class': 'cls_17',\n",
              "  'block_idx': 62,\n",
              "  'left': 420.55,\n",
              "  'level': 4,\n",
              "  'name': 'label',\n",
              "  'page_idx': 3,\n",
              "  'table_rows': [{'block_idx': 62,\n",
              "    'cells': [{'cell_value': '<s> A'},\n",
              "     {'cell_value': 'B'},\n",
              "     {'cell_value': 'C'},\n",
              "     {'cell_value': 'D'}],\n",
              "    'type': 'table_header'},\n",
              "   {'block_idx': 63,\n",
              "    'cell_value': 'Randomly Initialized Encoder',\n",
              "    'col_span': 4,\n",
              "    'type': 'full_row'}],\n",
              "  'tag': 'table',\n",
              "  'top': 109.04},\n",
              " {'block_class': 'cls_8',\n",
              "  'block_idx': 64,\n",
              "  'level': 3,\n",
              "  'page_idx': 3,\n",
              "  'sentences': ['A B C D E <s> A B C D E'],\n",
              "  'tag': 'header'},\n",
              " {'block_class': 'cls_18',\n",
              "  'block_idx': 65,\n",
              "  'level': 4,\n",
              "  'page_idx': 3,\n",
              "  'sentences': ['α', 'β', 'γ', 'δ ε'],\n",
              "  'tag': 'para'},\n",
              " {'block_class': 'cls_9',\n",
              "  'block_idx': 66,\n",
              "  'level': 5,\n",
              "  'page_idx': 3,\n",
              "  'sentences': ['(a) To use BART for classiﬁcation problems, the same input is fed into the encoder and decoder, and the repre- sentation from the ﬁnal output is used.',\n",
              "   '(b) For machine translation, we learn a small additional encoder that replaces the word embeddings in BART.',\n",
              "   'The new encoder can use a disjoint vocabulary.'],\n",
              "  'tag': 'list_item'},\n",
              " {'block_class': 'cls_7',\n",
              "  'block_idx': 67,\n",
              "  'level': 4,\n",
              "  'page_idx': 3,\n",
              "  'sentences': ['Figure 3: Fine tuning BART for classiﬁcation and translation.'],\n",
              "  'tag': 'para'},\n",
              " {'block_class': 'cls_7',\n",
              "  'block_idx': 68,\n",
              "  'level': 4,\n",
              "  'page_idx': 3,\n",
              "  'sentences': ['re-implement strong pre-training approaches recently proposed for discriminative and generation tasks.',\n",
              "   'We aim, as much as possible, to control for differences unrelated to the pre-training objective.',\n",
              "   'However, we do make minor changes to the learning rate and usage of layer normalisation in order to improve performance (tuning these separately for each objective).',\n",
              "   'For reference, we compare our implementations with published numbers from BERT, which was also trained for 1M steps on a combination of books and Wikipedia data.',\n",
              "   'We compare the following approaches:'],\n",
              "  'tag': 'para'},\n",
              " {'block_class': 'cls_7',\n",
              "  'block_idx': 69,\n",
              "  'level': 4,\n",
              "  'page_idx': 3,\n",
              "  'sentences': ['Language Model Similarly to GPT (Radford et al., 2018), we train a left-to-right Transformer language model.',\n",
              "   'This model is equivalent to the BART decoder, without cross-attention.'],\n",
              "  'tag': 'para'},\n",
              " {'block_class': 'cls_7',\n",
              "  'block_idx': 70,\n",
              "  'level': 4,\n",
              "  'page_idx': 3,\n",
              "  'sentences': ['Permuted Language Model Based on XLNet (Yang et al., 2019), we sample 1/6 of the tokens, and generate them in a random order autoregressively.',\n",
              "   'For consistency with other models, we do not implement the relative positional embeddings or attention across segments from XLNet.'],\n",
              "  'tag': 'para'},\n",
              " {'block_class': 'cls_7',\n",
              "  'block_idx': 71,\n",
              "  'level': 4,\n",
              "  'page_idx': 3,\n",
              "  'sentences': ['Masked Language Model Following BERT (Devlin et al., 2019), we replace 15% of tokens with [MASK] symbols, and train the model to independently predict the original tokens.'],\n",
              "  'tag': 'para'},\n",
              " {'block_class': 'cls_7',\n",
              "  'block_idx': 72,\n",
              "  'level': 4,\n",
              "  'page_idx': 3,\n",
              "  'sentences': ['Multitask Masked Language Model As in UniLM (Dong et al., 2019), we train a Masked Language Model with additional self-attention masks.',\n",
              "   'Self attention masks are chosen randomly in with the follow proportions: 1/6 left-to-right, 1/6 right-to-left, 1/3 unmasked, and 1/3 with the ﬁrst 50% of tokens unmasked and a left-to-right mask for the remainder.'],\n",
              "  'tag': 'para'},\n",
              " {'block_class': 'cls_7',\n",
              "  'block_idx': 73,\n",
              "  'level': 4,\n",
              "  'page_idx': 3,\n",
              "  'sentences': ['Masked Seq-to-Seq Inspired by MASS (Song et al., 2019), we mask a span containing 50% of tokens, and train a sequence to sequence model to predict the masked tokens.'],\n",
              "  'tag': 'para'},\n",
              " {'block_class': 'cls_7',\n",
              "  'block_idx': 74,\n",
              "  'level': 4,\n",
              "  'page_idx': 3,\n",
              "  'sentences': ['For the Permuted LM, Masked LM and Multitask Masked LM, we use two-stream attention (Yang et al., 2019) to efﬁciently compute likelihoods of the output part of the sequence (using a diagonal self-attention mask on the output to predict words left-to-right).'],\n",
              "  'tag': 'para'},\n",
              " {'block_class': 'cls_7',\n",
              "  'block_idx': 75,\n",
              "  'level': 4,\n",
              "  'page_idx': 3,\n",
              "  'sentences': ['We experiment with (1) treating the task as a standard sequence-to-sequence problem, where the source input to the encoder and the target is the decoder output, or (2) adding the source as preﬁx to the target in the decoder, with a loss only on the target part of the sequence.',\n",
              "   'We ﬁnd the former works better for BART models, and the latter for other models.'],\n",
              "  'tag': 'para'},\n",
              " {'block_class': 'cls_7',\n",
              "  'block_idx': 76,\n",
              "  'level': 4,\n",
              "  'page_idx': 3,\n",
              "  'sentences': ['To most directly compare our models on their ability to model their ﬁne-tuning objective (the log likelihood of the human text), we report perplexity in Table 1.'],\n",
              "  'tag': 'para'},\n",
              " {'block_class': 'cls_11',\n",
              "  'block_idx': 77,\n",
              "  'level': 3,\n",
              "  'page_idx': 3,\n",
              "  'sentences': ['4.2 Tasks'],\n",
              "  'tag': 'header'},\n",
              " {'block_class': 'cls_7',\n",
              "  'block_idx': 78,\n",
              "  'level': 5,\n",
              "  'page_idx': 3,\n",
              "  'sentences': ['SQuAD (Rajpurkar et al., 2016)a an extractive question answering task on Wikipedia paragraphs.',\n",
              "   'Answers are text spans extracted from a given document context.',\n",
              "   'Similar to BERT (Devlin et al., 2019), we use concatenated question and context as input to the encoder of BART, and additionally pass them to the decoder.',\n",
              "   'The model includes classiﬁers to predict the start and end indices of each token.'],\n",
              "  'tag': 'para'},\n",
              " {'block_class': 'cls_7',\n",
              "  'block_idx': 79,\n",
              "  'level': 5,\n",
              "  'page_idx': 3,\n",
              "  'sentences': ['MNLI (Williams et al., 2017), a bitext classiﬁcation task to predict whether one sentence entails another.',\n",
              "   'The ﬁne-tuned model concatenates the two sentences with appended an EOS token, and passes them to both the BART encoder and decoder.',\n",
              "   'In contrast to BERT, the representation of the EOS token is used to classify the sentences relations.'],\n",
              "  'tag': 'para'},\n",
              " {'block_class': 'cls_7',\n",
              "  'block_idx': 80,\n",
              "  'level': 5,\n",
              "  'page_idx': 3,\n",
              "  'sentences': ['ELI5 (Fan et al., 2019), a long-form abstractive question answering dataset.',\n",
              "   'Models generate answers conditioned on the concatenation of a question and supporting documents.'],\n",
              "  'tag': 'para'},\n",
              " {'block_class': 'cls_7',\n",
              "  'block_idx': 81,\n",
              "  'level': 5,\n",
              "  'page_idx': 3,\n",
              "  'sentences': ['XSum (Narayan et al., 2018), a news summarization dataset with highly abstractive summaries.'],\n",
              "  'tag': 'para'},\n",
              " {'block_class': 'cls_7',\n",
              "  'block_idx': 82,\n",
              "  'level': 5,\n",
              "  'page_idx': 3,\n",
              "  'sentences': ['ConvAI2 (Dinan et al., 2019), a dialogue response generation task, conditioned on context and a persona.'],\n",
              "  'tag': 'para'},\n",
              " {'block_class': 'cls_7',\n",
              "  'block_idx': 83,\n",
              "  'level': 5,\n",
              "  'page_idx': 3,\n",
              "  'sentences': ['CNN/DM (Hermann et al., 2015), a news summarization dataset.',\n",
              "   'Summaries here are typically closely related to source sentences.'],\n",
              "  'tag': 'para'},\n",
              " {'block_class': 'cls_11',\n",
              "  'block_idx': 84,\n",
              "  'level': 3,\n",
              "  'page_idx': 3,\n",
              "  'sentences': ['4.3 Results'],\n",
              "  'tag': 'header'},\n",
              " {'block_class': 'cls_7',\n",
              "  'block_idx': 85,\n",
              "  'level': 5,\n",
              "  'page_idx': 3,\n",
              "  'sentences': ['Results are shown in Table 1.', 'Several trends are clear:'],\n",
              "  'tag': 'para'},\n",
              " {'block_class': 'cls_11',\n",
              "  'block_idx': 86,\n",
              "  'left': 85.95,\n",
              "  'level': 4,\n",
              "  'name': '4.3 Results',\n",
              "  'page_idx': 4,\n",
              "  'table_rows': [{'block_idx': 86,\n",
              "    'cells': [{'cell_value': 'Model'},\n",
              "     {'cell_value': 'SQuAD 1.1'},\n",
              "     {'cell_value': 'MNLI'},\n",
              "     {'cell_value': 'ELI5'},\n",
              "     {'cell_value': 'XSum'},\n",
              "     {'cell_value': 'ConvAI2'},\n",
              "     {'cell_value': 'CNN/DM'}],\n",
              "    'type': 'table_header'},\n",
              "   {'block_idx': 87,\n",
              "    'cell_value': 'F1 Acc PPL PPL PPL PPL',\n",
              "    'col_span': 7,\n",
              "    'type': 'full_row'},\n",
              "   {'block_idx': 88,\n",
              "    'cells': [{'cell_value': 'BERT Base (Devlin et al., 2019)'},\n",
              "     {'cell_value': '88.5'},\n",
              "     {'cell_value': '84.3'},\n",
              "     {'cell_value': '-'},\n",
              "     {'cell_value': '-'},\n",
              "     {'cell_value': '-'},\n",
              "     {'cell_value': '-'}],\n",
              "    'type': 'table_data_row'},\n",
              "   {'block_idx': 89,\n",
              "    'cells': [{'cell_value': 'Masked Language Model'},\n",
              "     {'cell_value': '90.0'},\n",
              "     {'cell_value': '83.5'},\n",
              "     {'cell_value': '24.77'},\n",
              "     {'cell_value': '7.87'},\n",
              "     {'cell_value': '12.59'},\n",
              "     {'cell_value': '7.06'}],\n",
              "    'type': 'table_data_row'},\n",
              "   {'block_idx': 90,\n",
              "    'cells': [{'cell_value': 'Masked Seq2seq'},\n",
              "     {'cell_value': '87.0'},\n",
              "     {'cell_value': '82.1'},\n",
              "     {'cell_value': '23.40'},\n",
              "     {'cell_value': '6.80'},\n",
              "     {'cell_value': '11.43'},\n",
              "     {'cell_value': '6.19'}],\n",
              "    'type': 'table_data_row'},\n",
              "   {'block_idx': 91,\n",
              "    'cells': [{'cell_value': 'Language Model'},\n",
              "     {'cell_value': '76.7'},\n",
              "     {'cell_value': '80.1'},\n",
              "     {'cell_value': '21.40'},\n",
              "     {'cell_value': '7.00'},\n",
              "     {'cell_value': '11.51'},\n",
              "     {'cell_value': '6.56'}],\n",
              "    'type': 'table_data_row'},\n",
              "   {'block_idx': 92,\n",
              "    'cells': [{'cell_value': 'Permuted Language Model'},\n",
              "     {'cell_value': '89.1'},\n",
              "     {'cell_value': '83.7'},\n",
              "     {'cell_value': '24.03'},\n",
              "     {'cell_value': '7.69'},\n",
              "     {'cell_value': '12.23'},\n",
              "     {'cell_value': '6.96'}],\n",
              "    'type': 'table_data_row'},\n",
              "   {'block_idx': 93,\n",
              "    'cells': [{'cell_value': 'Multitask Masked Language Model'},\n",
              "     {'cell_value': '89.2'},\n",
              "     {'cell_value': '82.4'},\n",
              "     {'cell_value': '23.73'},\n",
              "     {'cell_value': '7.50'},\n",
              "     {'cell_value': '12.39'},\n",
              "     {'cell_value': '6.74'}],\n",
              "    'type': 'table_data_row'},\n",
              "   {'block_idx': 94,\n",
              "    'cells': [{'cell_value': 'BART Base w/ Token Masking'},\n",
              "     {'cell_value': '90.4'},\n",
              "     {'cell_value': '84.1'},\n",
              "     {'cell_value': '25.05'},\n",
              "     {'cell_value': '7.08'},\n",
              "     {'cell_value': '11.73'},\n",
              "     {'cell_value': '6.10'}],\n",
              "    'type': 'table_data_row'},\n",
              "   {'block_idx': 95,\n",
              "    'cells': [{'cell_value': 'w/ Token Deletion'},\n",
              "     {'cell_value': '90.4'},\n",
              "     {'cell_value': '84.1'},\n",
              "     {'cell_value': '24.61'},\n",
              "     {'cell_value': '6.90'},\n",
              "     {'cell_value': '11.46'},\n",
              "     {'cell_value': '5.87'}],\n",
              "    'type': 'table_data_row'},\n",
              "   {'block_idx': 96,\n",
              "    'cells': [{'cell_value': 'w/ Text Inﬁlling'},\n",
              "     {'cell_value': '90.8'},\n",
              "     {'cell_value': '84.0'},\n",
              "     {'cell_value': '24.26'},\n",
              "     {'cell_value': '6.61'},\n",
              "     {'cell_value': '11.05'},\n",
              "     {'cell_value': '5.83'}],\n",
              "    'type': 'table_data_row'},\n",
              "   {'block_idx': 97,\n",
              "    'cells': [{'cell_value': 'w/ Document Rotation'},\n",
              "     {'cell_value': '77.2'},\n",
              "     {'cell_value': '75.3'},\n",
              "     {'cell_value': '53.69'},\n",
              "     {'cell_value': '17.14'},\n",
              "     {'cell_value': '19.87'},\n",
              "     {'cell_value': '10.59'}],\n",
              "    'type': 'table_data_row'},\n",
              "   {'block_idx': 98,\n",
              "    'cells': [{'cell_value': 'w/ Sentence Shufﬂing'},\n",
              "     {'cell_value': '85.4'},\n",
              "     {'cell_value': '81.5'},\n",
              "     {'cell_value': '41.87'},\n",
              "     {'cell_value': '10.93'},\n",
              "     {'cell_value': '16.67'},\n",
              "     {'cell_value': '7.89'}],\n",
              "    'type': 'table_data_row'},\n",
              "   {'block_idx': 99,\n",
              "    'cells': [{'cell_value': 'w/ Text Inﬁlling + Sentence Shufﬂing'},\n",
              "     {'cell_value': '90.8'},\n",
              "     {'cell_value': '83.8'},\n",
              "     {'cell_value': '24.17'},\n",
              "     {'cell_value': '6.62'},\n",
              "     {'cell_value': '11.12'},\n",
              "     {'cell_value': '5.41'}],\n",
              "    'type': 'table_data_row'}],\n",
              "  'tag': 'table',\n",
              "  'top': 64.8},\n",
              " {'block_class': 'cls_7',\n",
              "  'block_idx': 100,\n",
              "  'level': 4,\n",
              "  'page_idx': 4,\n",
              "  'sentences': ['Table 1: Comparison of pre-training objectives.',\n",
              "   'All models are of comparable size and are trained for 1M steps on a combination of books and Wikipedia data.',\n",
              "   'Entries in the bottom two blocks are trained on identical data using the same code-base, and ﬁne-tuned with the same procedures.',\n",
              "   'Entries in the second block are inspired by pre-training objectives proposed in previous work, but have been simpliﬁed to focus on evaluation objectives (see §4.1).',\n",
              "   'Performance varies considerably across tasks, but the BART models with text inﬁlling demonstrate the most consistently strong performance.'],\n",
              "  'tag': 'para'},\n",
              " {'block_class': 'cls_7',\n",
              "  'block_idx': 101,\n",
              "  'level': 4,\n",
              "  'page_idx': 4,\n",
              "  'sentences': ['Performance of pre-training methods varies signiﬁcantly across tasks The effectiveness of pre-training methods is highly dependent on the task.',\n",
              "   'For example, a simple language model achieves the best ELI5 performance, but the worst SQUAD results.'],\n",
              "  'tag': 'para'},\n",
              " {'block_class': 'cls_7',\n",
              "  'block_idx': 102,\n",
              "  'level': 4,\n",
              "  'page_idx': 4,\n",
              "  'sentences': ['Token masking is crucial Pre-training objectives based on rotating documents or permuting sentences perform poorly in isolation.',\n",
              "   'The successful methods either use token deletion or masking, or self-attention masks.',\n",
              "   'Deletion appears to outperform masking on generation tasks.'],\n",
              "  'tag': 'para'},\n",
              " {'block_class': 'cls_7',\n",
              "  'block_idx': 103,\n",
              "  'level': 4,\n",
              "  'page_idx': 4,\n",
              "  'sentences': ['Left-to-right pre-training improves generation The Masked Language Model and the Permuted Language Model perform less well than others on generation, and are the only models we consider that do not include left-to-right auto-regressive language modelling during pre-training.'],\n",
              "  'tag': 'para'},\n",
              " {'block_class': 'cls_7',\n",
              "  'block_idx': 104,\n",
              "  'level': 4,\n",
              "  'page_idx': 4,\n",
              "  'sentences': ['Bidirectional encoders are crucial for SQuAD As noted in previous work (Devlin et al., 2019), just left-to-right decoder performs poorly on SQuAD, because future context is crucial in classiﬁcation decisions.',\n",
              "   'However, BART achieves similar performance with only half the number of bidirectional layers.'],\n",
              "  'tag': 'para'},\n",
              " {'block_class': 'cls_7',\n",
              "  'block_idx': 105,\n",
              "  'level': 4,\n",
              "  'page_idx': 4,\n",
              "  'sentences': ['The pre-training objective is not the only important factor Our Permuted Language Model performs less well than XLNet (Yang et al., 2019).',\n",
              "   'Some of this difference is likely due to not including other architectural improvements, such as relative-position embeddings or segment-level recurrence.'],\n",
              "  'tag': 'para'},\n",
              " {'block_class': 'cls_7',\n",
              "  'block_idx': 106,\n",
              "  'level': 4,\n",
              "  'page_idx': 4,\n",
              "  'sentences': ['Pure language models perform best on ELI5 The ELI5 dataset is an outlier, with much higher perplexities than other tasks, and is the only generation task where other models outperform BART.',\n",
              "   'A pure language model performs best, suggesting that BART is less effective when the output is only loosely constrained by the input.'],\n",
              "  'tag': 'para'},\n",
              " {'block_class': 'cls_7',\n",
              "  'block_idx': 107,\n",
              "  'level': 4,\n",
              "  'page_idx': 4,\n",
              "  'sentences': ['BART achieves the most consistently strong performance.',\n",
              "   'With the exception of ELI5, BART models using text-inﬁlling perform well on all tasks.'],\n",
              "  'tag': 'para'},\n",
              " {'block_class': 'cls_1',\n",
              "  'block_idx': 108,\n",
              "  'level': 2,\n",
              "  'page_idx': 4,\n",
              "  'sentences': ['5 Large-scale Pre-training Experiments'],\n",
              "  'tag': 'header'},\n",
              " {'block_class': 'cls_7',\n",
              "  'block_idx': 109,\n",
              "  'level': 3,\n",
              "  'page_idx': 4,\n",
              "  'sentences': ['Recent work has shown that downstream performance can dramatically improve when pre-training is scaled to large batch sizes (Yang et al., 2019; Liu et al., 2019) and corpora.',\n",
              "   'To test how well BART performs in this regime, and to create a useful model for downstream tasks, we trained BART using the same scale as the RoBERTa model.'],\n",
              "  'tag': 'para'},\n",
              " {'block_class': 'cls_11',\n",
              "  'block_idx': 110,\n",
              "  'level': 3,\n",
              "  'page_idx': 4,\n",
              "  'sentences': ['5.1 Experimental Setup'],\n",
              "  'tag': 'header'},\n",
              " {'block_class': 'cls_7',\n",
              "  'block_idx': 111,\n",
              "  'level': 3,\n",
              "  'page_idx': 4,\n",
              "  'sentences': ['We pre-train a large model with 12 layers in each of the encoder and decoder, and a hidden size of 1024.',\n",
              "   'Following RoBERTa (Liu et al., 2019), we use a batch size of 8000, and train the model for 500000 steps.',\n",
              "   'Documents are tokenized with the same byte-pair encoding as GPT-2 (Radford et al., 2019).',\n",
              "   'Based on the results in Section §4, we use a combination of text inﬁlling and sentence permutation.',\n",
              "   'We mask 30% of tokens in each document, and permute all sentences.',\n",
              "   'Although sentence permutation only shows signiﬁcant additive gains'],\n",
              "  'tag': 'para'},\n",
              " {'block_class': 'cls_11',\n",
              "  'block_idx': 112,\n",
              "  'left': 130.05,\n",
              "  'level': 3,\n",
              "  'name': '5.1 Experimental Setup',\n",
              "  'page_idx': 5,\n",
              "  'table_rows': [{'block_idx': 112,\n",
              "    'cells': [{'cell_value': ''},\n",
              "     {'cell_value': 'SQuAD 1.1 EM/F1'},\n",
              "     {'cell_value': 'SQuAD 2.0 EM/F1'},\n",
              "     {'cell_value': 'MNLI m/mm'},\n",
              "     {'cell_value': 'SST Acc'},\n",
              "     {'cell_value': 'QQP Acc'},\n",
              "     {'cell_value': 'QNLI Acc'},\n",
              "     {'cell_value': 'STS-B Acc'},\n",
              "     {'cell_value': 'RTE Acc'},\n",
              "     {'cell_value': 'MRPC Acc'},\n",
              "     {'cell_value': 'CoLA Mcc'}],\n",
              "    'type': 'table_header'},\n",
              "   {'block_idx': 113,\n",
              "    'cells': [{'cell_value': 'BERT'},\n",
              "     {'cell_value': '84.1/90.9'},\n",
              "     {'cell_value': '79.0/81.8'},\n",
              "     {'cell_value': '86.6/-'},\n",
              "     {'cell_value': '93.2'},\n",
              "     {'cell_value': '91.3'},\n",
              "     {'cell_value': '92.3'},\n",
              "     {'cell_value': '90.0'},\n",
              "     {'cell_value': '70.4'},\n",
              "     {'cell_value': '88.0'},\n",
              "     {'cell_value': '60.6'}],\n",
              "    'type': 'table_data_row'},\n",
              "   {'block_idx': 114,\n",
              "    'cells': [{'cell_value': 'UniLM'},\n",
              "     {'cell_value': '-/-'},\n",
              "     {'cell_value': '80.5/83.4'},\n",
              "     {'cell_value': '87.0/85.9'},\n",
              "     {'cell_value': '94.5'},\n",
              "     {'cell_value': '-'},\n",
              "     {'cell_value': '92.7'},\n",
              "     {'cell_value': '-'},\n",
              "     {'cell_value': '70.9'},\n",
              "     {'cell_value': '-'},\n",
              "     {'cell_value': '61.1'}],\n",
              "    'type': 'table_data_row'},\n",
              "   {'block_idx': 115,\n",
              "    'cells': [{'cell_value': 'XLNet'},\n",
              "     {'cell_value': '89.0/94.5'},\n",
              "     {'cell_value': '86.1/88.8'},\n",
              "     {'cell_value': '89.8/-'},\n",
              "     {'cell_value': '95.6'},\n",
              "     {'cell_value': '91.8'},\n",
              "     {'cell_value': '93.9'},\n",
              "     {'cell_value': '91.8'},\n",
              "     {'cell_value': '83.8'},\n",
              "     {'cell_value': '89.2'},\n",
              "     {'cell_value': '63.6'}],\n",
              "    'type': 'table_data_row'},\n",
              "   {'block_idx': 116,\n",
              "    'cells': [{'cell_value': 'RoBERTa'},\n",
              "     {'cell_value': '88.9/94.6'},\n",
              "     {'cell_value': '86.5/89.4'},\n",
              "     {'cell_value': '90.2/90.2'},\n",
              "     {'cell_value': '96.4'},\n",
              "     {'cell_value': '92.2'},\n",
              "     {'cell_value': '94.7'},\n",
              "     {'cell_value': '92.4'},\n",
              "     {'cell_value': '86.6'},\n",
              "     {'cell_value': '90.9'},\n",
              "     {'cell_value': '68.0'}],\n",
              "    'type': 'table_data_row'},\n",
              "   {'block_idx': 117,\n",
              "    'cells': [{'cell_value': 'BART'},\n",
              "     {'cell_value': '88.8/94.6'},\n",
              "     {'cell_value': '86.1/89.2'},\n",
              "     {'cell_value': '89.9/90.1'},\n",
              "     {'cell_value': '96.6'},\n",
              "     {'cell_value': '92.5'},\n",
              "     {'cell_value': '94.9'},\n",
              "     {'cell_value': '91.2'},\n",
              "     {'cell_value': '87.0'},\n",
              "     {'cell_value': '90.4'},\n",
              "     {'cell_value': '62.8'}],\n",
              "    'type': 'table_data_row'}],\n",
              "  'tag': 'table',\n",
              "  'top': 64.8},\n",
              " {'block_class': 'cls_7',\n",
              "  'block_idx': 118,\n",
              "  'level': 3,\n",
              "  'page_idx': 5,\n",
              "  'sentences': ['Table 2: Results for large models on SQuAD and GLUE tasks.',\n",
              "   'BART performs comparably to RoBERTa and XLNet, suggesting that BART’s uni-directional decoder layers do not reduce performance on discriminative tasks.'],\n",
              "  'tag': 'para'},\n",
              " {'block_class': 'cls_11',\n",
              "  'block_idx': 119,\n",
              "  'left': 305.7,\n",
              "  'level': 4,\n",
              "  'name': '5.1 Experimental Setup',\n",
              "  'page_idx': 5,\n",
              "  'table_rows': [{'block_idx': 119,\n",
              "    'cells': [{'cell_value': '', 'col_span': 1},\n",
              "     {'cell_value': 'CNN/DailyMail', 'col_span': 3},\n",
              "     {'cell_value': 'XSum', 'col_span': 2}],\n",
              "    'type': 'table_header'},\n",
              "   {'block_idx': 120,\n",
              "    'cells': [{'cell_value': ''},\n",
              "     {'cell_value': 'R1'},\n",
              "     {'cell_value': 'R2'},\n",
              "     {'cell_value': 'RL'},\n",
              "     {'cell_value': 'R1'},\n",
              "     {'cell_value': 'R2'},\n",
              "     {'cell_value': 'RL'}],\n",
              "    'type': 'table_header'},\n",
              "   {'block_idx': 121,\n",
              "    'cells': [{'cell_value': 'Lead-3'},\n",
              "     {'cell_value': '40.42'},\n",
              "     {'cell_value': '17.62'},\n",
              "     {'cell_value': '36.67'},\n",
              "     {'cell_value': '16.30'},\n",
              "     {'cell_value': '1.60'},\n",
              "     {'cell_value': '11.95'}],\n",
              "    'type': 'table_data_row'},\n",
              "   {'block_idx': 122,\n",
              "    'cells': [{'cell_value': 'PTGEN (See et al., 2017)'},\n",
              "     {'cell_value': '36.44'},\n",
              "     {'cell_value': '15.66'},\n",
              "     {'cell_value': '33.42'},\n",
              "     {'cell_value': '29.70'},\n",
              "     {'cell_value': '9.21'},\n",
              "     {'cell_value': '23.24'}],\n",
              "    'type': 'table_data_row'},\n",
              "   {'block_idx': 123,\n",
              "    'cells': [{'cell_value': 'PTGEN+COV (See et al., 2017)'},\n",
              "     {'cell_value': '39.53'},\n",
              "     {'cell_value': '17.28'},\n",
              "     {'cell_value': '36.38'},\n",
              "     {'cell_value': '28.10'},\n",
              "     {'cell_value': '8.02'},\n",
              "     {'cell_value': '21.72'}],\n",
              "    'type': 'table_data_row'},\n",
              "   {'block_idx': 124,\n",
              "    'cells': [{'cell_value': 'UniLM'},\n",
              "     {'cell_value': '43.33'},\n",
              "     {'cell_value': '20.21'},\n",
              "     {'cell_value': '40.51'},\n",
              "     {'cell_value': '-'},\n",
              "     {'cell_value': '-'},\n",
              "     {'cell_value': '-'}],\n",
              "    'type': 'table_data_row'},\n",
              "   {'block_idx': 125,\n",
              "    'cells': [{'cell_value': 'BERTSUMABS (Liu & Lapata, 2019)'},\n",
              "     {'cell_value': '41.72'},\n",
              "     {'cell_value': '19.39'},\n",
              "     {'cell_value': '38.76'},\n",
              "     {'cell_value': '38.76'},\n",
              "     {'cell_value': '16.33'},\n",
              "     {'cell_value': '31.15'}],\n",
              "    'type': 'table_data_row'},\n",
              "   {'block_idx': 126,\n",
              "    'cells': [{'cell_value': 'BERTSUMEXTABS (Liu & Lapata, 2019)'},\n",
              "     {'cell_value': '42.13'},\n",
              "     {'cell_value': '19.60'},\n",
              "     {'cell_value': '39.18'},\n",
              "     {'cell_value': '38.81'},\n",
              "     {'cell_value': '16.50'},\n",
              "     {'cell_value': '31.27'}],\n",
              "    'type': 'table_data_row'},\n",
              "   {'block_idx': 127,\n",
              "    'cells': [{'cell_value': 'BART'},\n",
              "     {'cell_value': '44.16'},\n",
              "     {'cell_value': '21.28'},\n",
              "     {'cell_value': '40.90'},\n",
              "     {'cell_value': '45.14'},\n",
              "     {'cell_value': '22.27'},\n",
              "     {'cell_value': '37.25'}],\n",
              "    'type': 'table_data_row'}],\n",
              "  'tag': 'table',\n",
              "  'top': 205.41},\n",
              " {'block_class': 'cls_7',\n",
              "  'block_idx': 128,\n",
              "  'level': 3,\n",
              "  'page_idx': 5,\n",
              "  'sentences': ['Table 3: Results on two standard summarization datasets.',\n",
              "   'BART outperforms previous work on summarization on two tasks and all metrics, with gains of roughly 6 points on the more abstractive dataset.'],\n",
              "  'tag': 'para'},\n",
              " {'block_class': 'cls_7',\n",
              "  'block_idx': 129,\n",
              "  'level': 3,\n",
              "  'page_idx': 5,\n",
              "  'sentences': ['on the CNN/DM summarization dataset, we hypothesised that larger pre-trained models may be better able to learn from this task.',\n",
              "   'To help the model better ﬁt the data, we disabled dropout for the ﬁnal 10% of training steps.',\n",
              "   'We use the same pre-training data as Liu et al.',\n",
              "   '(2019), consisting of 160Gb of news, books, stories, and web text.'],\n",
              "  'tag': 'para'},\n",
              " {'block_class': 'cls_11',\n",
              "  'block_idx': 130,\n",
              "  'level': 3,\n",
              "  'page_idx': 5,\n",
              "  'sentences': ['5.2 Discriminative Tasks'],\n",
              "  'tag': 'header'},\n",
              " {'block_class': 'cls_7',\n",
              "  'block_idx': 131,\n",
              "  'level': 5,\n",
              "  'page_idx': 5,\n",
              "  'sentences': ['Table 2 compares the performance of BART with several recent approaches on the well-studied SQuAD and GLUE tasks (Warstadt et al., 2018; Socher et al., 2013; Dolan & Brockett, 2005; Agirre et al., 2007; Williams et al., 2018; Dagan et al., 2006; Levesque et al., 2011).'],\n",
              "  'tag': 'para'},\n",
              " {'block_class': 'cls_7',\n",
              "  'block_idx': 132,\n",
              "  'level': 5,\n",
              "  'page_idx': 5,\n",
              "  'sentences': ['The most directly comparable baseline is RoBERTa, which was pre-trained with the same resources, but a different objective.',\n",
              "   'Overall, BART performs similarly, with only small differences between the models on most tasks.',\n",
              "   'suggesting that BART’s improvements on generation tasks do not come at the expense of classiﬁcation performance.'],\n",
              "  'tag': 'para'},\n",
              " {'block_class': 'cls_11',\n",
              "  'block_idx': 133,\n",
              "  'level': 3,\n",
              "  'page_idx': 5,\n",
              "  'sentences': ['5.3 Generation Tasks'],\n",
              "  'tag': 'header'},\n",
              " {'block_class': 'cls_7',\n",
              "  'block_idx': 134,\n",
              "  'level': 5,\n",
              "  'page_idx': 5,\n",
              "  'sentences': ['We also experiment with several text generation tasks.',\n",
              "   'BART is ﬁne-tuned as a standard sequence-to-sequence model from the input to the output text.',\n",
              "   'During ﬁnetuning we use a label smoothed cross entropy loss (Pereyra et al., 2017), with the smoothing parameter set to 0.1.',\n",
              "   'During generation, we set beam size as 5, remove duplicated trigrams in beam search, and tuned the model with min-len, max-len, length penalty on the validation set (Fan et al., 2017).'],\n",
              "  'tag': 'para'},\n",
              " {'block_class': 'cls_7',\n",
              "  'block_idx': 135,\n",
              "  'left': 414.62,\n",
              "  'level': 6,\n",
              "  'name': '5.3 Generation Tasks',\n",
              "  'page_idx': 5,\n",
              "  'table_rows': [{'block_idx': 135,\n",
              "    'cells': [{'cell_value': '', 'col_span': 1},\n",
              "     {'cell_value': 'ConvAI2', 'col_span': 2},\n",
              "     {'cell_value': 'Valid F1 Valid PPL', 'col_span': 0}],\n",
              "    'type': 'table_header'},\n",
              "   {'block_idx': 136,\n",
              "    'cells': [{'cell_value': 'Seq2Seq + Attention'},\n",
              "     {'cell_value': '16.02'},\n",
              "     {'cell_value': '35.07'}],\n",
              "    'type': 'table_header'},\n",
              "   {'block_idx': 137,\n",
              "    'cells': [{'cell_value': 'Best System'},\n",
              "     {'cell_value': '19.09'},\n",
              "     {'cell_value': '17.51'}],\n",
              "    'type': 'table_data_row'},\n",
              "   {'block_idx': 138,\n",
              "    'cells': [{'cell_value': 'BART'},\n",
              "     {'cell_value': '20.72'},\n",
              "     {'cell_value': '11.85'}],\n",
              "    'type': 'table_data_row'}],\n",
              "  'tag': 'table',\n",
              "  'top': 382.91},\n",
              " {'block_class': 'cls_7',\n",
              "  'block_idx': 139,\n",
              "  'level': 5,\n",
              "  'page_idx': 5,\n",
              "  'sentences': ['Table 4: BART outperforms previous work on conversational response generation.',\n",
              "   'Perplexities are renormalized based on ofﬁcial tokenizer for ConvAI2.'],\n",
              "  'tag': 'para'},\n",
              " {'block_class': 'cls_7',\n",
              "  'block_idx': 140,\n",
              "  'level': 5,\n",
              "  'page_idx': 5,\n",
              "  'sentences': ['Summarization To provide a comparison with the state-of-the-art in summarization, we present results on two summarization datasets, CNN/DailyMail and XSum, which have distinct properties.'],\n",
              "  'tag': 'para'},\n",
              " {'block_class': 'cls_7',\n",
              "  'block_idx': 141,\n",
              "  'level': 5,\n",
              "  'page_idx': 5,\n",
              "  'sentences': ['Summaries in the CNN/DailyMail tend to resemble source sentences.',\n",
              "   'Extractive models do well here, and even the baseline of the ﬁrst-three source sentences is highly competitive.',\n",
              "   'Nevertheless, BART outperforms all existing work.'],\n",
              "  'tag': 'para'},\n",
              " {'block_class': 'cls_7',\n",
              "  'block_idx': 142,\n",
              "  'level': 5,\n",
              "  'page_idx': 5,\n",
              "  'sentences': ['In contrast, XSum is highly abstractive, and extractive models perform poorly.',\n",
              "   'BART outperforms the best previous work, which leverages BERT, by roughly 6.0 points on all ROUGE metrics—representing a signiﬁcant advance in performance on this problem.',\n",
              "   'Qualitatively, sample quality is high (see §6).'],\n",
              "  'tag': 'para'},\n",
              " {'block_class': 'cls_7',\n",
              "  'block_idx': 143,\n",
              "  'level': 5,\n",
              "  'page_idx': 5,\n",
              "  'sentences': ['Dialogue We evaluate dialogue response generation on CONVAI2 (Dinan et al., 2019), in which agents must generate responses conditioned on both the previous context and a textually-speciﬁed persona.',\n",
              "   'BART outperforms previous work on two automated metrics.'],\n",
              "  'tag': 'para'},\n",
              " {'block_class': 'cls_7',\n",
              "  'block_idx': 144,\n",
              "  'level': 3,\n",
              "  'page_idx': 6,\n",
              "  'sentences': ['R2 RL'],\n",
              "  'tag': 'header'},\n",
              " {'block_class': 'cls_7',\n",
              "  'block_idx': 145,\n",
              "  'left': 98.03,\n",
              "  'level': 4,\n",
              "  'name': 'R2 RL',\n",
              "  'page_idx': 6,\n",
              "  'table_rows': [{'block_idx': 145,\n",
              "    'cells': [{'cell_value': 'Best Extractive'},\n",
              "     {'cell_value': '23.5'},\n",
              "     {'cell_value': '3.1'},\n",
              "     {'cell_value': '17.5'}],\n",
              "    'type': 'table_header'},\n",
              "   {'block_idx': 146,\n",
              "    'cells': [{'cell_value': 'Language Model Seq2Seq'},\n",
              "     {'cell_value': '27.8 28.3'},\n",
              "     {'cell_value': '4.7 5.1'},\n",
              "     {'cell_value': '23.1 22.8'}],\n",
              "    'type': 'table_data_row'},\n",
              "   {'block_idx': 147,\n",
              "    'cells': [{'cell_value': 'Seq2Seq Multitask BART'},\n",
              "     {'cell_value': '28.9 30.6'},\n",
              "     {'cell_value': '5.4 6.2'},\n",
              "     {'cell_value': '23.1 24.3'}],\n",
              "    'type': 'table_data_row'}],\n",
              "  'tag': 'table',\n",
              "  'top': 93.71},\n",
              " {'block_class': 'cls_7',\n",
              "  'block_idx': 148,\n",
              "  'level': 4,\n",
              "  'page_idx': 6,\n",
              "  'sentences': ['Table 5: BART achieves state-of-the-art results on the challenging ELI5 abstractive question answering dataset.',\n",
              "   'Comparison models are from Fan et al.',\n",
              "   '(2019).'],\n",
              "  'tag': 'para'},\n",
              " {'block_class': 'cls_7',\n",
              "  'block_idx': 149,\n",
              "  'level': 4,\n",
              "  'page_idx': 6,\n",
              "  'sentences': ['RO-EN'],\n",
              "  'tag': 'header'},\n",
              " {'block_class': 'cls_7',\n",
              "  'block_idx': 150,\n",
              "  'left': 133.58,\n",
              "  'level': 5,\n",
              "  'name': 'RO-EN',\n",
              "  'page_idx': 6,\n",
              "  'tag': 'table',\n",
              "  'top': 234.47},\n",
              " {'block_class': 'cls_1',\n",
              "  'block_idx': 161,\n",
              "  'left': 307.28,\n",
              "  'level': 0,\n",
              "  'name': 'RO-EN',\n",
              "  'page_idx': 6,\n",
              "  'table_rows': [{'block_idx': 150,\n",
              "    'cells': [{'cell_value': 'Baseline'}, {'cell_value': '36.80'}],\n",
              "    'type': 'table_data_row'},\n",
              "   {'block_idx': 151,\n",
              "    'cells': [{'cell_value': 'Fixed BART 36.29 Tuned BART 37.96'}],\n",
              "    'type': 'table_data_row'},\n",
              "   {'block_idx': 152,\n",
              "    'cells': [{'cell_value': 'Table 6: The performance (BLEU) of baseline and BART on WMT’16 RO-EN augmented with backtranslation data. BART improves over a strong backtranslation (BT) baseline by using monolingual English pre-training.'}],\n",
              "    'type': 'table_data_row'},\n",
              "   {'block_idx': 153,\n",
              "    'cells': [{'cell_value': 'Abstractive QA We use the recently proposed ELI5 dataset to test the model’s ability to generate long freeform answers. We ﬁnd BART outperforms the best previous work by 1.2 ROUGE-L, but the dataset remains a challenging, because answers are only weakly speciﬁed by the question.'}],\n",
              "    'type': 'table_data_row'},\n",
              "   {'block_idx': 154,\n",
              "    'cells': [{'cell_value': '5.4 Translation'}],\n",
              "    'type': 'table_data_row'},\n",
              "   {'block_idx': 155,\n",
              "    'cells': [{'cell_value': 'We also evaluated performance on WMT16 RomanianEnglish, augmented with back-translation data from Sennrich et al. (2016). We use a 6-layer transformer source encoder to map Romanian into a representation that BART is able to de-noise into English, following the approach introduced in §3.4. Experiment results are presented in Table 6. We compare our results against a baseline Transformer architecture (Vaswani et al., 2017) with Transformerlarge settings (the baseline row). We show the performance of both steps of our model in the ﬁxed BART and tuned BART rows. For each row we experiment on the original WMT16 Romanian-English augmented with back-translation data. We use a beam width of 5 and a length penalty of α = 1. Preliminary results suggested that our approach was less effective without back-translation data, and prone to overﬁtting—future work should explore additional regularization techniques.'}],\n",
              "    'type': 'table_data_row'},\n",
              "   {'block_idx': 156,\n",
              "    'cells': [{'cell_value': '6 Qualitative Analysis'}],\n",
              "    'type': 'table_data_row'},\n",
              "   {'block_idx': 157,\n",
              "    'cells': [{'cell_value': 'BART shows large improvements on summarization metrics, of up to 6 points over the prior state-of-the-art. To understand BART’s performance beyond automated metrics, we analyse its generations qualitatively.'}],\n",
              "    'type': 'table_data_row'},\n",
              "   {'block_idx': 158,\n",
              "    'cells': [{'cell_value': 'Table 7 shows example summaries generated by BART. Examples are taken from WikiNews articles published after the creation of the pre-training corpus, to eliminate the possibility of the events described being present in the model’s training data. Following Narayan et al. (2018), we remove the ﬁrst sentence of the article prior to summarizing it, so there is no easy extractive summary of the document.'}],\n",
              "    'type': 'table_data_row'},\n",
              "   {'block_idx': 159,\n",
              "    'cells': [{'cell_value': 'Unsurprisingly, model output is ﬂuent and grammatical English. However, model output is also highly abstractive, with few phrases copied from the input. The output is also generally factually accurate, and integrates supporting evidence from across the input document with background knowledge (for example, correctly completing names, or inferring that PG&E operates in California). In the ﬁrst example, inferring that ﬁsh are protecting reefs from global warming requires non-trivial inference from the text. However, the claim that the work was published in Science is not supported by the source.'}],\n",
              "    'type': 'table_data_row'},\n",
              "   {'block_idx': 160,\n",
              "    'cells': [{'cell_value': 'These samples demonstrate that the BART pretraining has learned a strong combination of natural language understanding and generation.'}],\n",
              "    'type': 'table_data_row'},\n",
              "   {'block_idx': 161,\n",
              "    'cells': [{'cell_value': '7 Related Work'}, {'cell_value': ''}],\n",
              "    'type': 'table_header'},\n",
              "   {'block_idx': 162,\n",
              "    'cell_value': 'Early methods for pretraining were based on language models. GPT (Radford et al., 2018) only models left- ward context, which is problematic for some tasks. ELMo (Peters et al., 2018) concatenates left-only and right-only representations, but does not pre-train inter- actions between these features. Radford et al. (2019) demonstrated that very large language models can act as unsupervised multitask models.',\n",
              "    'col_span': 2,\n",
              "    'type': 'full_row'},\n",
              "   {'block_idx': 163,\n",
              "    'cell_value': 'BERT (Devlin et al., 2019) introduced masked lan- guage modelling, which allows pre-training to learn in- teractions between left and right context words. Re- cent work has shown that very strong performance can be achieved by training for longer (Liu et al., 2019), by tying parameters across layers (Lan et al., 2019), and by masking spans instead of words (Joshi et al., 2019). Predictions are not made auto-regressively, re- ducing the effectiveness of BERT for generation tasks.',\n",
              "    'col_span': 2,\n",
              "    'type': 'full_row'},\n",
              "   {'block_idx': 164,\n",
              "    'cell_value': 'UniLM (Dong et al., 2019) ﬁne-tunes BERT with an ensemble of masks, some of which allow only leftward context. Like BART, this allows UniLM to be used for both generative and discriminative tasks. A difference is that UniLM predictions are conditionally indepen- dent, whereas BART’s are autoregressive. BART re- duces the mismatch between pre-training and genera- tion tasks, because the decoder is always trained on un- corrupted context.',\n",
              "    'col_span': 2,\n",
              "    'type': 'full_row'},\n",
              "   {'block_idx': 165,\n",
              "    'cell_value': 'MASS (Song et al., 2019) is perhaps the most similar model to BART. An input sequence where a contiguous span of tokens is masked is mapped to a sequence con- sisting of the missing tokens. MASS is less effective for discriminative tasks, because disjoint sets of tokens are fed into the encoder and decoder.',\n",
              "    'col_span': 2,\n",
              "    'type': 'full_row'},\n",
              "   {'block_idx': 166,\n",
              "    'cell_value': 'XL-Net (Yang et al., 2019) extends BERT by pre-',\n",
              "    'col_span': 2,\n",
              "    'type': 'full_row'},\n",
              "   {'block_idx': 167,\n",
              "    'cells': [{'cell_value': 'Source Document (abbreviated)'},\n",
              "     {'cell_value': 'BART Summary'}],\n",
              "    'type': 'table_data_row'},\n",
              "   {'block_idx': 168,\n",
              "    'cells': [{'cell_value': 'The researchers examined three types of coral in reefs off the coast of Fiji The researchers found when ﬁsh were plentiful, they would eat algae and seaweed off the corals, which appeared to leave them more resistant to the bacterium Vibrio coralliilyti- cus, a bacterium associated with bleaching. The researchers sug- gested the algae, like warming temperatures, might render the corals’ chemical defenses less effective, and the ﬁsh were pro- tecting the coral by removing the algae.'},\n",
              "     {'cell_value': 'Fisheries off the coast of Fiji are protect- ing coral reefs from the effects of global warming, according to a study in the jour- nal Science.'}],\n",
              "    'type': 'table_data_row'},\n",
              "   {'block_idx': 169,\n",
              "    'cells': [{'cell_value': 'Sacoolas, who has immunity as a diplomat’s wife, was involved in a trafﬁc collision Prime Minister Johnson was questioned about the case while speaking to the press at a hospital in Wat- ford. He said, “I hope that Anne Sacoolas will come back if we can’t resolve it then of course I will be raising it myself personally with the White House.”'},\n",
              "     {'cell_value': 'Boris Johnson has said he will raise the is- sue of US diplomat Anne Sacoolas’ diplo- matic immunity with the White House.'}],\n",
              "    'type': 'table_data_row'},\n",
              "   {'block_idx': 170,\n",
              "    'cells': [{'cell_value': 'According to Syrian state media, government forces began de- ploying into previously SDF controlled territory yesterday. On October 6, US President Donald Trump and Turkish Presi- dent Recep Tayyip Erdoan spoke on the phone. Then both na- tions issued statements speaking of an imminent incursion into northeast Syria . On Wednesday, Turkey began a military offensive with airstrikes followed by a ground invasion. Syrian government forces have entered territory held by the US-backed Syrian Democratic Forces (SDF) in response to Turkey’s incursion into the region.'},\n",
              "     {'cell_value': ''}],\n",
              "    'type': 'table_data_row'},\n",
              "   {'block_idx': 171,\n",
              "    'cells': [{'cell_value': 'This is the ﬁrst time anyone has been recorded to run a full marathon of 42.195 kilometers (approximately 26 miles) under this pursued landmark time. It was not, however, an ofﬁcially sanctioned world record, as it was not an”open race” of the IAAF. His time was 1 hour 59 minutes 40.2 seconds. Kipchoge ran in Vienna, Austria. It was an event speciﬁcally designed to help Kipchoge break the two hour barrier.'},\n",
              "     {'cell_value': 'Kenyan runner Eliud Kipchoge has run a marathon in less than two hours.'}],\n",
              "    'type': 'table_data_row'},\n",
              "   {'block_idx': 172,\n",
              "    'cells': [{'cell_value': 'PG&E stated it scheduled the blackouts in response to forecasts for high winds amid dry conditions. The aim is to reduce the risk of wildﬁres. Nearly 800 thousand customers were scheduled to be affected by the shutoffs which were expected to last through at least midday tomorrow.'},\n",
              "     {'cell_value': 'Power has been turned off to millions of customers in California as part of a power shutoff plan.'}],\n",
              "    'type': 'table_data_row'}],\n",
              "  'tag': 'table',\n",
              "  'top': 348.11},\n",
              " {'block_class': 'cls_7',\n",
              "  'block_idx': 173,\n",
              "  'level': 0,\n",
              "  'page_idx': 7,\n",
              "  'sentences': ['Table 7: Example summaries from the XSum-tuned BART model on WikiNews articles.',\n",
              "   'For clarity, only relevant excerpts of the source are shown.',\n",
              "   'Summaries combine information from across the article and prior knowledge.'],\n",
              "  'tag': 'para'},\n",
              " {'block_class': 'cls_7',\n",
              "  'block_idx': 174,\n",
              "  'level': 0,\n",
              "  'page_idx': 7,\n",
              "  'sentences': ['dicting masked tokens auto-regressively in a permuted order.',\n",
              "   'This objective allows predictions to condition on both left and right context.',\n",
              "   'In contrast, the BART decoder works left-to-right during pre-training, matching the setting during generation.'],\n",
              "  'tag': 'para'},\n",
              " {'block_class': 'cls_7',\n",
              "  'block_idx': 175,\n",
              "  'level': 0,\n",
              "  'page_idx': 7,\n",
              "  'sentences': ['Several papers have explored using pre-trained representations to improve machine translation.',\n",
              "   'The largest improvements have come from pre-training on both source and target languages (Song et al., 2019; Lample & Conneau, 2019), but this requires pretraining on all languages of interest.',\n",
              "   'Other work has shown that encoders can be improved using pre-trained representations (Edunov et al., 2019), but gains in decoders are more limited.',\n",
              "   'We show how BART can be used to improve machine translation decoders.'],\n",
              "  'tag': 'para'},\n",
              " {'block_class': 'cls_1',\n",
              "  'block_idx': 176,\n",
              "  'level': 0,\n",
              "  'page_idx': 7,\n",
              "  'sentences': ['8 Conclusions'],\n",
              "  'tag': 'header'},\n",
              " {'block_class': 'cls_7',\n",
              "  'block_idx': 177,\n",
              "  'level': 1,\n",
              "  'page_idx': 7,\n",
              "  'sentences': ['introduced BART, a pre-training approach that learns to map corrupted documents to the original.',\n",
              "   'BART achieves similar performance to RoBERTa on discriminative tasks, while achieving new state-of-theart results on a number of text generation tasks.',\n",
              "   'Future work should explore new methods for corrupting documents for pre-training, perhaps tailoring them to speciﬁc end tasks.'],\n",
              "  'tag': 'para'},\n",
              " {'block_class': 'cls_1',\n",
              "  'block_idx': 178,\n",
              "  'level': 0,\n",
              "  'page_idx': 8,\n",
              "  'sentences': ['References'],\n",
              "  'tag': 'header'},\n",
              " {'block_class': 'cls_7',\n",
              "  'block_idx': 179,\n",
              "  'level': 1,\n",
              "  'page_idx': 8,\n",
              "  'sentences': ['Eneko Agirre, Llu’is M‘arquez, and Richard Wicentowski (eds.).',\n",
              "   'Proceedings of the Fourth International Workshop on Semantic Evaluations (SemEval2007).',\n",
              "   'Association for Computational Linguistics, Prague, Czech Republic, June 2007.'],\n",
              "  'tag': 'para'},\n",
              " {'block_class': 'cls_7',\n",
              "  'block_idx': 180,\n",
              "  'level': 1,\n",
              "  'page_idx': 8,\n",
              "  'sentences': ['Ido Dagan, Oren Glickman, and Bernardo Magnini.',\n",
              "   'The PASCAL recognising textual entailment challenge.',\n",
              "   'In Machine learning challenges.',\n",
              "   'evaluating predictive uncertainty, visual object classiﬁcation, and recognising tectual entailment, pp.',\n",
              "   '177– 190.',\n",
              "   'Springer, 2006.'],\n",
              "  'tag': 'para'},\n",
              " {'block_class': 'cls_7',\n",
              "  'block_idx': 181,\n",
              "  'level': 1,\n",
              "  'page_idx': 8,\n",
              "  'sentences': ['Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova.',\n",
              "   'BERT: Pre-training of deep bidirectional transformers for language understanding.',\n",
              "   'In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pp.',\n",
              "   '4171– 4186, Minneapolis, Minnesota, June 2019.',\n",
              "   'Association for Computational Linguistics.',\n",
              "   'doi: 10.18653/ v1/N19-1423.',\n",
              "   'URL https://www.aclweb.',\n",
              "   'org/anthology/N19-1423.'],\n",
              "  'tag': 'para'},\n",
              " {'block_class': 'cls_7',\n",
              "  'block_idx': 182,\n",
              "  'level': 1,\n",
              "  'page_idx': 8,\n",
              "  'sentences': ['Emily Dinan, Varvara Logacheva, Valentin Malykh, Alexander Miller, Kurt Shuster, Jack Urbanek, Douwe Kiela, Arthur Szlam, Iulian Serban, Ryan Lowe, et al.',\n",
              "   'The second conversational intelligence challenge (convai2).',\n",
              "   'arXiv preprint arXiv:1902.00098, 2019.'],\n",
              "  'tag': 'para'},\n",
              " {'block_class': 'cls_7',\n",
              "  'block_idx': 183,\n",
              "  'level': 1,\n",
              "  'page_idx': 8,\n",
              "  'sentences': ['William B Dolan and Chris Brockett.',\n",
              "   'Automatically constructing a corpus of sentential paraphrases.',\n",
              "   'In Proceedings of the International Workshop on Paraphrasing, 2005.'],\n",
              "  'tag': 'para'},\n",
              " {'block_class': 'cls_7',\n",
              "  'block_idx': 184,\n",
              "  'level': 1,\n",
              "  'page_idx': 8,\n",
              "  'sentences': ['Li Dong, Nan Yang, Wenhui Wang, Furu Wei, Xiaodong Liu, Yu Wang, Jianfeng Gao, Ming Zhou, and Hsiao-Wuen Hon.',\n",
              "   'Uniﬁed language model pretraining for natural language understanding and generation.',\n",
              "   'arXiv preprint arXiv:1905.03197, 2019.'],\n",
              "  'tag': 'para'},\n",
              " {'block_class': 'cls_14',\n",
              "  'block_idx': 185,\n",
              "  'level': 1,\n",
              "  'page_idx': 8,\n",
              "  'sentences': ['Sergey Edunov, Alexei Baevski, and Michael Auli.',\n",
              "   'Pre-trained language model representations for language generation.',\n",
              "   'In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), 2019.'],\n",
              "  'tag': 'para'},\n",
              " {'block_class': 'cls_7',\n",
              "  'block_idx': 186,\n",
              "  'level': 1,\n",
              "  'page_idx': 8,\n",
              "  'sentences': ['Angela Fan, David Grangier, and Michael Auli.',\n",
              "   'Controllable abstractive summarization.',\n",
              "   'arXiv preprint arXiv:1711.05217, 2017.'],\n",
              "  'tag': 'para'},\n",
              " {'block_class': 'cls_7',\n",
              "  'block_idx': 187,\n",
              "  'level': 1,\n",
              "  'page_idx': 8,\n",
              "  'sentences': ['Angela Fan, Yacine Jernite, Ethan Perez, David Grangier, Jason Weston, and Michael Auli.',\n",
              "   'Eli5: Long form question answering.',\n",
              "   'arXiv preprint arXiv:1907.09190, 2019.'],\n",
              "  'tag': 'para'},\n",
              " {'block_class': 'cls_7',\n",
              "  'block_idx': 188,\n",
              "  'level': 1,\n",
              "  'page_idx': 8,\n",
              "  'sentences': ['Dan Hendrycks and Kevin Gimpel.',\n",
              "   'Gaussian error linear units (gelus).',\n",
              "   'arXiv preprint arXiv:1606.08415, 2016.'],\n",
              "  'tag': 'para'},\n",
              " {'block_class': 'cls_7',\n",
              "  'block_idx': 189,\n",
              "  'level': 1,\n",
              "  'page_idx': 8,\n",
              "  'sentences': ['Karl Moritz Hermann, Tomas Kocisky, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom.',\n",
              "   'Teaching machines to read and comprehend.',\n",
              "   'In Advances in neural information processing systems, pp.',\n",
              "   '1693–1701, 2015.'],\n",
              "  'tag': 'para'},\n",
              " {'block_class': 'cls_7',\n",
              "  'block_idx': 190,\n",
              "  'level': 1,\n",
              "  'page_idx': 8,\n",
              "  'sentences': ['Mandar Joshi, Danqi Chen, Yinhan Liu, Daniel S Weld, Luke Zettlemoyer, and Omer Levy.',\n",
              "   'Spanbert: Improving pre-training by representing and predicting spans.',\n",
              "   'arXiv preprint arXiv:1907.10529, 2019.'],\n",
              "  'tag': 'para'},\n",
              " {'block_class': 'cls_7',\n",
              "  'block_idx': 191,\n",
              "  'level': 1,\n",
              "  'page_idx': 8,\n",
              "  'sentences': ['Guillaume Lample and Alexis Conneau.',\n",
              "   'Crosslingual language model pretraining.',\n",
              "   'arXiv preprint arXiv:1901.07291, 2019.'],\n",
              "  'tag': 'para'},\n",
              " {'block_class': 'cls_7',\n",
              "  'block_idx': 192,\n",
              "  'level': 1,\n",
              "  'page_idx': 8,\n",
              "  'sentences': ['Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut.',\n",
              "   'Albert: A lite bert for self-supervised learning of language representations.',\n",
              "   'arXiv preprint arXiv:1909.11942, 2019.'],\n",
              "  'tag': 'para'},\n",
              " {'block_class': 'cls_7',\n",
              "  'block_idx': 193,\n",
              "  'level': 1,\n",
              "  'page_idx': 8,\n",
              "  'sentences': ['Hector J Levesque, Ernest Davis, and Leora Morgenstern.',\n",
              "   'The Winograd schema challenge.',\n",
              "   'In AAAI Spring Symposium: Logical Formalizations of Commonsense Reasoning, volume 46, pp.',\n",
              "   '47, 2011.'],\n",
              "  'tag': 'para'},\n",
              " {'block_class': 'cls_7',\n",
              "  'block_idx': 194,\n",
              "  'level': 1,\n",
              "  'page_idx': 8,\n",
              "  'sentences': ['Yang Liu and Mirella Lapata.',\n",
              "   'Text summarization with pretrained encoders.',\n",
              "   'arXiv preprint arXiv:1908.08345, 2019.'],\n",
              "  'tag': 'para'},\n",
              " {'block_class': 'cls_7',\n",
              "  'block_idx': 195,\n",
              "  'level': 1,\n",
              "  'page_idx': 8,\n",
              "  'sentences': ['Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov.',\n",
              "   'Roberta: A robustly optimized bert pretraining approach.'],\n",
              "  'tag': 'para'},\n",
              " {'block_class': 'cls_14',\n",
              "  'block_idx': 196,\n",
              "  'level': 1,\n",
              "  'page_idx': 8,\n",
              "  'sentences': ['arXiv preprint arXiv:1907.11692, 2019.'],\n",
              "  'tag': 'para'},\n",
              " {'block_class': 'cls_7',\n",
              "  'block_idx': 197,\n",
              "  'level': 1,\n",
              "  'page_idx': 8,\n",
              "  'sentences': ['Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean.',\n",
              "   'Efﬁcient estimation of word representations in vector space.',\n",
              "   'arXiv preprint arXiv:1301.3781, 2013.'],\n",
              "  'tag': 'para'},\n",
              " {'block_class': 'cls_7',\n",
              "  'block_idx': 198,\n",
              "  'level': 1,\n",
              "  'page_idx': 8,\n",
              "  'sentences': ['Shashi Narayan, Shay B Cohen, and Mirella Lapata.',\n",
              "   'Don’t give me the details, just the summary!',\n",
              "   'topicaware convolutional neural networks for extreme summarization.',\n",
              "   'arXiv preprint arXiv:1808.08745, 2018.'],\n",
              "  'tag': 'para'},\n",
              " {'block_class': 'cls_7',\n",
              "  'block_idx': 199,\n",
              "  'level': 1,\n",
              "  'page_idx': 8,\n",
              "  'sentences': ['Gabriel Pereyra, George Tucker, Jan Chorowski, Łukasz Kaiser, and Geoffrey Hinton.',\n",
              "   'Regularizing neural networks by penalizing conﬁdent output distributions.',\n",
              "   'arXiv preprint arXiv:1701.06548, 2017.'],\n",
              "  'tag': 'para'},\n",
              " {'block_class': 'cls_7',\n",
              "  'block_idx': 200,\n",
              "  'level': 1,\n",
              "  'page_idx': 8,\n",
              "  'sentences': ['Matthew E Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer.',\n",
              "   'Deep contextualized word representations.',\n",
              "   'arXiv preprint arXiv:1802.05365, 2018.'],\n",
              "  'tag': 'para'},\n",
              " {'block_class': 'cls_7',\n",
              "  'block_idx': 201,\n",
              "  'level': 1,\n",
              "  'page_idx': 8,\n",
              "  'sentences': ['Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever.',\n",
              "   'Improving language understanding by generative pre-training.',\n",
              "   'URL https://s3-us-west-2.',\n",
              "   'amazonaws.',\n",
              "   'com/openaiassets/researchcovers/languageunsupervised/language understanding paper.',\n",
              "   'pdf, 2018.'],\n",
              "  'tag': 'para'},\n",
              " {'block_class': 'cls_7',\n",
              "  'block_idx': 202,\n",
              "  'level': 1,\n",
              "  'page_idx': 8,\n",
              "  'sentences': ['Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever.',\n",
              "   'Language models are unsupervised multitask learners.',\n",
              "   'OpenAI Blog, 1(8), 2019.'],\n",
              "  'tag': 'para'},\n",
              " {'block_class': 'cls_7',\n",
              "  'block_idx': 203,\n",
              "  'level': 1,\n",
              "  'page_idx': 9,\n",
              "  'sentences': ['Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang.',\n",
              "   'Squad: 100,000+ questions for machine comprehension of text.',\n",
              "   'arXiv preprint arXiv:1606.05250, 2016.'],\n",
              "  'tag': 'para'},\n",
              " {'block_class': 'cls_7',\n",
              "  'block_idx': 204,\n",
              "  'level': 1,\n",
              "  'page_idx': 9,\n",
              "  'sentences': ['Abigail See, Peter J Liu, and Christopher D Manning.',\n",
              "   'Get to the point: Summarization with pointer-generator networks.',\n",
              "   'arXiv preprint arXiv:1704.04368, 2017.'],\n",
              "  'tag': 'para'},\n",
              " {'block_class': 'cls_7',\n",
              "  'block_idx': 205,\n",
              "  'level': 1,\n",
              "  'page_idx': 9,\n",
              "  'sentences': ['Rico Sennrich, Barry Haddow, and Alexandra Birch.',\n",
              "   'Edinburgh neural machine translation systems for WMT 16.',\n",
              "   'In Proceedings of the First Conference on Machine Translation: Volume 2, Shared Task Papers, 2016.'],\n",
              "  'tag': 'para'},\n",
              " {'block_class': 'cls_7',\n",
              "  'block_idx': 206,\n",
              "  'level': 1,\n",
              "  'page_idx': 9,\n",
              "  'sentences': ['Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D Manning, Andrew Ng, and Christopher Potts.',\n",
              "   'Recursive deep models for semantic compositionality over a sentiment treebank.',\n",
              "   'In Proceedings of EMNLP, pp.',\n",
              "   '1631–1642, 2013.'],\n",
              "  'tag': 'para'},\n",
              " {'block_class': 'cls_7',\n",
              "  'block_idx': 207,\n",
              "  'level': 1,\n",
              "  'page_idx': 9,\n",
              "  'sentences': ['Kaitao Song, Xu Tan, Tao Qin, Jianfeng Lu, and TieYan Liu.',\n",
              "   'Mass: Masked sequence to sequence pretraining for language generation.',\n",
              "   'In International Conference on Machine Learning, 2019.'],\n",
              "  'tag': 'para'},\n",
              " {'block_class': 'cls_7',\n",
              "  'block_idx': 208,\n",
              "  'level': 1,\n",
              "  'page_idx': 9,\n",
              "  'sentences': ['Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin.',\n",
              "   'Attention is all you need.',\n",
              "   'In Advances in neural information processing systems, pp.',\n",
              "   '5998–6008, 2017.'],\n",
              "  'tag': 'para'},\n",
              " {'block_class': 'cls_7',\n",
              "  'block_idx': 209,\n",
              "  'level': 1,\n",
              "  'page_idx': 9,\n",
              "  'sentences': ['Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman.',\n",
              "   'Glue: A multi-task benchmark and analysis platform for natural language understanding.',\n",
              "   'arXiv preprint arXiv:1804.07461, 2018.'],\n",
              "  'tag': 'para'},\n",
              " {'block_class': 'cls_7',\n",
              "  'block_idx': 210,\n",
              "  'level': 1,\n",
              "  'page_idx': 9,\n",
              "  'sentences': ['Alex Warstadt, Amanpreet Singh, and Samuel R. Bowman.',\n",
              "   'Neural network acceptability judgments.'],\n",
              "  'tag': 'para'},\n",
              " {'block_class': 'cls_14',\n",
              "  'block_idx': 211,\n",
              "  'level': 1,\n",
              "  'page_idx': 9,\n",
              "  'sentences': ['arXiv preprint 1805.12471, 2018.'],\n",
              "  'tag': 'para'},\n",
              " {'block_class': 'cls_7',\n",
              "  'block_idx': 212,\n",
              "  'level': 1,\n",
              "  'page_idx': 9,\n",
              "  'sentences': ['Adina Williams, Nikita Nangia, and Samuel R Bowman.',\n",
              "   'A broad-coverage challenge corpus for sentence understanding through inference.',\n",
              "   'arXiv preprint arXiv:1704.05426, 2017.'],\n",
              "  'tag': 'para'},\n",
              " {'block_class': 'cls_7',\n",
              "  'block_idx': 213,\n",
              "  'level': 1,\n",
              "  'page_idx': 9,\n",
              "  'sentences': ['Adina Williams, Nikita Nangia, and Samuel R. Bowman.',\n",
              "   'A broad-coverage challenge corpus for sentence understanding through inference.',\n",
              "   'In Proceedings of NAACL-HLT, 2018.'],\n",
              "  'tag': 'para'},\n",
              " {'block_class': 'cls_7',\n",
              "  'block_idx': 214,\n",
              "  'level': 1,\n",
              "  'page_idx': 9,\n",
              "  'sentences': ['Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Ruslan Salakhutdinov, and Quoc V Le.',\n",
              "   'Xlnet: Generalized autoregressive pretraining for language understanding.',\n",
              "   'arXiv preprint arXiv:1906.08237, 2019.'],\n",
              "  'tag': 'para'}]"
            ]
          },
          "metadata": {},
          "execution_count": 17
        }
      ]
    }
  ]
}